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ABSTRACT . . . 

* . A three-year pro ject. was conducted in Pennsylvania to 

design and implement a comprehensive model evaluation management 
information and policy support unit within the state vocational 
rehabilitation (VR) agency. Five mandated project tasks were carried "\ 
out to produce products that are necessary for program evaluation ' 
(PE) . Proposed Federal Evaluation' Standards were' field -tested; the 
functions, of the£ PE section ftere conceptualized; ' techniques and 
methods of PE 'were developed and personnel were trained for PE; 
utilization tfechjj&iques, were developed and improved; and various' 
resources were used 'to* assist in qualitative review'. At the 
completion, of the three-year contract, 10 products Were produced for 
use by a model VR PE unit. These - products include- case review * ' 

studies, an annual plan fpr case reviews* client, fibllow^up studies,, a- 
computerized system and procedures for identifying and reporting 
dollars saved .through use- of noiiVBVR resources, a technical 
assistance .center, a profile .analysis, technique to be used as a 
management tool,, a data base "system, a program evaulation se'cti-on* 
organization handbook, a management" information system, and 
guidelines for development of program evaluation plans. Evaluation of 
the project led to recommendations for refinements and recognition of 
.the. difficulty of the evaluation .task".. (The report is a'narrative on 
ongoing activities, -rather than a.conceptual review of the project.) 
.( KC ) ' . . 
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; ' s . , ~ FOREWORD . 
In October 1978 the Program .Evaluation Section ofi^the Tennsylvania 
Bureau of Vocational Rehabilitation was warded a three-year* federal 1 
contract entitled "Comprehensive State 'VR Program Polity Systems through 
Model Evaluation/ Management "Information Support Unit". The goal \<yE this 
contract wa^to develop mc/del^state program evaluation unit dimensions > 
under a variety 5 of circumstances, as a prelude to,' and providing, models 
for later nationwide* implementation of %wluat ion- capacity . ' . .« 



•Th^Ls document was written not' from a conceptual point' of view but 

rather as a practical day-to-day view, of what happened and how it happ^Sd'. 

It is our hope that this report will assist other agencies to enhance their 

f current activities and develop new evaluation capacities. / 

If -additional information is desired on any aspect^yfs this contract, 

* • * 

please contact: Mr. Harry W. Guise, Admiriistrator , program* Evaluaticpn 

f * 

Section, Bureau of . Vocational Rehabilitation, 1318 Labor and Industry 
Builpmg, H^rrisburg,. Pennsylvania 17120. Telephone: (717) 787-7312. ' 
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I. Introduction . * ( 

Base'd upon-national competitive bidding, Pennsylvania,, on " • t 
October 1, 1978, was awarded a three-year contract entitled "Compre- 
hensive -State VR Program and Policy Systems throilgh Model Evaluation/ 

Management Information Support Unit» !t ' 

" r 
. < Criteria for the six states awarv^d the contract a>re as follows^ 

1. One .agency for the visually impaired^ Mississippi Blind 

/Agency — . 

2» 'Five general. 6r/combined (general or blind) agencies in the 

following categories.:* t • 

* « * 

A. Two "Small states (Section 1J.0 budget for Fiscal Year 1378 - 

rf under $10 million) - Delaware and Oregon 
. B» Two medium-siz^d states .(Section 110 budget for Fiscal 
Year \ 978 - between $10 billion and $25 million) - i 
Virginia and .Michigan / * 

C. One large-sized state (Section 110 budget for Fiscal 

Year 1978 ^over $25 million) - Pennsylvania 

w The legislative authority for developing this contract comes '\ 
. V ' - ' 

from Section ^01. of the Rehabilitation- Act of 1973 which mandates the 
" e • * 

Secretary of^Health, Education and Welf/re to measure and evaluate 

4 i 

the impact of all programs authorised by this Act in order to determine 
their effectiveness in* achieving stated goalls in general and their impact 
on programs. 



II. PURPOSES AND OBJECTIVES OF THE CONTRACT 
A.. PURPOSE \ ' " 

— i — * » 

To design and implement a comprehensive model evaluation management 

information and policy support unit within six state vocational 
* * . — - * \ * 

* * i * 

rehabilitation (VR) agencies. The model units will use evaluation data- 

-to support the processes of management policy an4 decision making, 

* ♦ * " * 

budgeting, program planning and development and program operations 

** ' * ' ' 

through direct linkages with the VR units responsible for those functions . 

These goals will be accomplished through the staffing of an evaluation 

unit with the ability to meet the objectives of: A) utilizing existing 

Rehabilitation Services Administration (RSA) directives and instruments; 

B) field testing and developing new RSA evaluation- instruments ana , 

evaluation .propesses C) developing additional evaluation crapacity in 

the state; agency program evaluation uijit to 'go beyond traditional 

reporting particularly with' regard to measuring client change resulting 

from the provision of rehabilitation services, and D) the capacity to 

respond, through development of relevant -processes and projects," 

including revisions to information systems, to evaluation needs parti- 

cular to the state agency. To accomplish these goals it is expected 

the evaluation unit, in connection with other administrative an£ 

program units, will develop the capability to assess management policy 

and information needs and provide evaluation data in a form and context 

appropriate to the identified needs. 

The goal of .RSA in this contract is to develop, model state evaluation 

unit dimensions^ under a variety of circumstances, as a prelude to, and 

providing mpdels foia later nationwide implementation^: evaluation 

capacity. The project is expected to* demonstrate that given sufficient 

resources, state agencies, regardless of size ^ can develop effective ' 



evaluation systems. Model evaluation units wiJJ^ be established for 

each .of" the following administrative structures: A) one agency for the 

visually impaired; B) one umbrella agency, and C) three general agencies - 

on the basis of size: small,- medium, and large. Pennsylvania was 

• $ 

awarded the large statfe contract iir October 1978. ' 
B . > PROJECT OBJECTIVES " ■ 

* 4 

'1. . To develop a model of a comprehensive program and policy system. linked 
by. appropriate evaluation data which is suitahle for various types 
of state VR evaluation units. ' 4 . • % / 
A. To develop a strong Situation unit that is capafcle of assessing 
management infestation needs- and, provide information to manage-: 
ment. The t information provided needs to relate to the efficiency 
of agency administrative processes, the achievement of agency 
objectives, and the effects services have on clients. 



£. * To put the state agency % program evaluation unit in an organi 

zational -( administrative) position so Vt is directly 'responsible 
. and reports directly to /the agency administrator. 
'C. To employ in the mode]*, unit, personnel able^to: 1) conceptualize 
agency ^information needs, plans and mea surable o bjectives and 

v , upe*. them in evaluation studies; 2) conduct evaluation surveys and 
other studies; 3) work with data processing; 4) analyze data; 
'5) create ^a^uat^pp reports' of great utility, and 6) work in 
pre-planned areas with agency personnel to assure the development 
a^nd dissemination of evaluation information. 

D. ' to Conduct studies pf. particular interest, to individual states 

beyond data required for Federal reporting purposes. 

E. - To develop a conceptual framework for program evaluation within 

the Vftlagency. ^ Central .to program and project evaluation^ is a 



. , two-year evaluation .plan for the state agency which is updated ^ 
. * annually. * " 

^r. To. achieve an evaluation unit that is open-ende$U and flexible. 
This will encourage o the 'integration and/or development of neW 
" evaluation technology. . ' 

2. , To field test and evaluate the effectiveness for state VR agency 

management of the revised standards for evaluation, and the facilities 

* *- . : . 

* » • 

information 'system'. The implementation of the fac^ities infor- 1 
mation system will be for the purpose of evaluating its effective- 
ness in potentially assisting and managing resources related to 
all medical and vocational facilities within the state from whom 
there is a certain level of purchase of service, 

3. To build new evaluation capacity which can be generalized , to other 

states' such as the development of indicators of intermediate and 

end-process client outcomes,' with related measureSy, to describe * f 

• * 
client change resulting from the provision of rehabilitation 

services (competency based client outcome evaluation). To* implement 

into the state VR program new developments in evaluation capacity 

sucfcjais t&e rehabilitation, indicators, weighted case closure, cost- 

benefit analysis and functional assessment scales, and other related 

projects . , % 4 

» '• * ,» 

4. To develop linkages for an"internal (within^ th.e~agency) an<\an , t 

•7 ' . \ . V' / 

* external (between state agencies^ communication, dissemination and 
utilization network. ' • * 

A. "To provide a pre-planned level of technical consultation on' 
evaluation topics,. This will be for five purposes: ^ 
1. 'To determine program and information needs and* special 
study requirements tfy examination^ bf the rehabilitation 



* • v 
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process and consultation with agency personnel. 

2. Tb conduct special studies (e.g., needs* of the severely 
disabled, effectiveness of rehabilitation services) . and 
by- bringing these, to. administrative attention to help 
-improve the agency 1 s services impact. > 

3. To revise, as needed, and based upon thorough study, 

the state agency information system to support the program 

' ' - ' / 

evaluatic^n^/ planning-, development , budgeting, program 
implementation and monitoring processes. 

4. To have central' and regional offices 1 staffs fully , aware 
of the. utility anci potential of the model evaluation unit 
through Report irfg' of plans and progress. • 

5. To bring, within two years, the degree of competency in 
program evaluation units to the planned level*. 

To develop an evaluation plan' for the model unit for the purpose 
of guiding and monitoring the degree to which the objectives of . 
the program evaluation unit and contractual objectives and tasks 
are met . 

To estimate the manpower arid budget needs of a model evaluation 
unit within tfie state agency, to indicate the state's ability and 
interest in maintaining the model unit beyond the contract period, 
and to indicate the objectives and other requirements of the model 
unit with a task implementation and timetable plan which will be 
regularly tracked by the State VR agency administrator.. 

, r- ~ 

f 
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III. CONTRACT TASKS AND ACCOMPLISHMENTS 



A. OVERVIEW 

The three years of MEU activity were marked by satisfaction and 

frustration. For Pennsylvania,* Task 6 (Development of New Evaluation 

Capacity) was the most fruitful in terms of assisting Pennsylvania BVR. 

The entire Program Evaluation Section benefitted from the organized 

response necessary to respond to RFP 1 s , deliverables, >ad hoc MEU 

«■ * • 

\ \ 

information requests, and dev^b^ment and refinement of contract products. 

This document was written not from a conceptue^goint of view but 

rather a practical day-to-day view of what happened and l?ow it happened. 

The staff persons most involved d'id the writing. As^a result, this 

effort is stretighj^forward and to the point.. , r % , , r ^ / 

' i. ! ^ ^ " * 

As you review this Finai^iReport , the satisf ac^icm^/ and disappoint- 

ments will b€9®S' very ciearf , .Qverlfa the thre^year contract had a very 

positive effect on the Pennsylvania BVR in general and the Pennsylvania 

I * < 

Program -^valuation Section in particular. 

5 / 

B. ' CONTkACJT TASKS 



The ^contract activities center aroun^numerous tasks. These tasks 

are mandated objectives that wefe developec^and implemented to produce 
*' ' 4 7 ^ t 

products that are necessary ifi the utilization of program evaluation (PE) 

The tasks relate to the following PE work activities: ^ * 

1. On'e task was*concerned with ^ield testing of proposed Federal 

Evaluatidn Standards to be implemented on a« national basis on 

October 1, 1982. The Federal Evaluation Standards will megfeuv& y 

*tie efficiency effectiveness, and impact of the rehabilitation 

p:?ogram\ / , * 

2\ Three tasks were primarily concerned with conceptualization 

of the functions in our PE Section, monitoring arid aspects of * 



managing the project, 

^Two ta^ks relate to refinement and 'development of methods 
* * * 

.and techniques of PE. Also development of PE ^organization 

and personnel capable of: J ' ^ 

* 

A. Interpreting' Agency information needs, plans and measurable 
objectives . , r k 

B\ n Conduct PE surveys and "other studies 

C. "Analyze data , % 

D. . .Work knowledgeably with £ata processing and information 

system , * 

E. '* Create PE reports of great utility 

» 

F. Work in, pre-planned areas with Agency personnel to assure 

** development, dissemination and utilization of PE information 
» f 

One task deals with improvement and development of dissemination 
and utilization techniques using the following as guidelines : * 
.A. Needs sensing « 

B. ^ Product development *^ 

C. Dissemination 

D. Utilization * * . « 

E. Capacity building , . t 

The e task of project evaluation subjects each task to qualitative 

# 1 

reviews utilizing various "Resources . v 

' . *** 

The following "have been. utilized to assist in the development 
and implementation of these tasks: v ; 

"A. Rehabilitation Services Administration 

B. - Federal Regional Offjtee 

C. Private contractors 

% 

D. Other states ♦ 
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E. Rehabilitation Research Institutes' 

F. Research and Training Centers 

G. BVfr field staff . > 

•PRODl/fcTS AT COMPLETION OF THE THREE-YEAR CONTRACT 



Case Review Studies - Cag^Revlew Process is an individual case 
review concerned with the case service patterns within the 
rehabilitation process and provides information for use i*n program 
monitoring. * 
This system was developed and is undergoing further refinement at 
both the administrative and technical levels. 

Annual Plap for Case Reviews - This plan defines and details functions 
and 'respgnsibili ties for conducting case reviews during a^one-year 
period^ * < ' 

Clie nt Follow-up Studies - . - 

A. Current Federal Prog ram Evaluation Standards - retention of client 
benefits * 

B. Current Federal Program Evaluation Standards* - Client's satis- 
faction with PA-BVR services * , 

C. Life, Satisfaction Survey - Client's satisfaction with PA-BVR 
services/client ! £ satisfaction with life 

Similar Benefits - Design of a computerized system and procedures 
for identifying and reporting the estimated dollars' saved through 
the use*of non-BVR resources as they relateXo our clients. 
Technical Assistance Center - This Center has been developed and 



is being maintained by the Program .Evaluation Section, to assist 
PA-BVR personnel at all levels with technical assistance and. 
information needs as they relate to all phases of vocational rehabi- 
litation in the 'delivery of services to^ clients* 



6 . Profile .Analysis Technique - This management tool which is currently 

Y? ■ 

being* used by our District, Regional, and Central Office staff to 
monit6r the delivery of services to clients is being- refined and' 

A ' 4 ' 

.expanded frpm an outcome -variable profile to a process, referral, 
and outcome variable profile. 

7. Facility Data* Base System - This involves the development of a 



CO 



mputerized reporting system on all PA-BVR* clients receivings 
services in rehabilitation facilities. Currently the^e is no 
information system to measure the performance of facilities as it 
relates to ofir clitents. 

8 . Program Eva luation Section Organization Handbook - A major review 
~* — 7 ; 

of the Program Evaluation Section has been completed with the 
development of a handbook that outlines organization, functions 
and assignments . This management tc>ol will be used by the 
Administrator of Evaluation to monitor the activities within his 
Section* / * 

9. . Management Information System -* The System has the fqll6wing 
functions: \ 

A. Assessment to determine information needs 

B. , To determine i'f appropriate audience is receiving appropriate 

information ' 

C. To c^etermiTie^if information is utilized and how V 

D. To determine methodology to be used in developing M.I.S. / 

1. 'A review and assessment of Central Office utilization of 

PE Section products has been completed. 
2^ Two other phases involving District and Regional Office 
v staff 



utilization of products will be ^ompleted in the 
final year. The final product will be a directory of 
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1 * * 

technical assistance available from the PE Section, " # 

10, Development of Program Evaluation Plans' - Since all Special projects 

now have a, program evaluation component, the PE Section h^s^esigned 

a pre-plan for .the Facilities and Grants Section's new projects on 

Client Assistance and Independent Living . 

Each year of the three-year contract period has. resulted in the 

continuous refinement and improvement c5f the products, •.Products 

i . r - 

f developed and implemented prior to the contract were: 

'l, Case Review Process . • 

2, Follow-up Studj.es # 

3, ProVile Analysis 
D. PA MEU -DEVELOPMENT PHILOSOPHY 

In 'developing a Model- one of our, preliminary concerns was , 

to structure this Unit in such a manner that it would have the greatest! 
possible impact on the Agency's policy-jnaking, planning, and program ~" • 
operations. The RFP required that the Unit look particularly at the 
'.'direct linkages" it had with the sections responsible for each of 
these functions within *he Agency. InVttempting to -situate the ModeJ. 
Unit within the Agency's organization and' to develop a "model" structure* 
it seemed important to consider the- characteristics ofithe State Agency 
in- terms of its organizational structyre and hierarchy. The Pennsylvania 
Agency was awarded this contract in the category of a large^ agency 
/ (Section A'10 fund over $25 million) and, 'with itfs size, the Agency ^ 

possesses the. characteristics xof a large bureaucracy. The literature 
■ • on' organizational theory and structure' (Porter, et al , 1975) suggests 

that a large organization attempts to maintain its existing state of 
"'.equilibrium. Although we would argue that the Pennsylvania -Agency is 
remarkably fluid and dynamic for its size, i«t must necessarily possess • 

ERIC . 



some degree of this characteristic.. Even though change could have 
been initiated in the organization through revolutionary methods, the A 
existing Program Evaluation Unit as well as the prevailing management 
philosophy showed a clear* preference for an evolutionary style of . 
impact. This logically requires that, the source's initiating change 
be present over some significant degree of time: Therefore , durability 
4s considered .to be a key element in establishing this Unit and 



-periocj 



structuring its functions so they would continue after the contract 



In reviewing the RFP, ijt was clear that the % f ederal government 
intended for the Unit to continue after the^contract, and in fact 

required a commitment from the Agency that it vould do so.~ Despite 

> 

this condition to the contract award, as a result of economic changes 

^that have occurred recently, there now appears to be some threat to 

the continuation bf Program Evaluation both at the state and national 

levels. As Taylor (1979) indicated in his brief review of the develop- 

ment of Program. Evaluation, this area is relatively new "for rehabilitation 

agencies. Although the 1966 Amendments to the Rehabilitation Act are 

probably the first precursors of program evaluation in vocational 

rehabilitation, the 1973 Amendments clearly established this function 

withiiv, m<^st agencies'. However, problems with the J'edesal Prograji 

Standards, the ambiguity of mandating legislation as well as defen- 

saveness of jnany managers across the country, has led to continuing 

♦ . 

resjrstence to its 'operation. The latest legislation no longer specifically 
~. „ 

identifies program eyaluatioSf as a required component of VR operations N 

& ' , * 
and leaves its place ambiguous. When combined with the austerity of low 

' ' - f , • > ' ' ' 

^tate budgets at this time and the forecast for even greater cutbacks 

in the future, the utility and cost-effectiveness of program evaluation 



within; state agencies is being clo^el\r scrutinized* \ Duration or., 
even survivability becomes a gei^nane issue for all program evaluation, 
and it especially relevant for a model unit. However, this crisis 
also offers exciting opportunities for the model to clearly prove 
its utility and to demonstrate an ability to improve operations and 
make them more efficient in such plvcfta^,times^ # * 

From-t^ie beginning it was felt that the Model Unites Impact* and 
chances of its work being durable would be substantially increased 
if it could be closely tied to thfe existing Program Evalua^on Section. 

Several other factors were designed to insure the. Unit's impact 
and durability. If a unit is to be accepted within an existing organi- 
zation it should immediately begin 'to produce work that is of importance 
to the agency's recognized concerns and neWs. In this instance, 
several pro jects -were selected that required field studies , .and \^he Unit 1 
Case Service Eva^uators were immediately put to work on them so th§kt 
products were available within several months after the contractus initi- 
ation. In addition, ^selecting ^a Unit staff experienced in our Program; 
Evaluation techniques and Agency operations was also useful in moving * 

rapidly to an operational capacity. $ • v • 

K r 

Our experiences m constructing the Model Evaluation Unit suggest 

• > ' • „.« S ',. 

that* this process was considerably more .cliff icult and time consuming than 
had been previously thought. Tki particular, the need to fqcus on the 
design and structural characteristics of the Unit, bpth to increase its 
durability and to experimentally fine} the best organizational 'structure , 
required more conceptualization organizational support, and participant* 
commitment and flexibility thati was anticipated. The payoff to both 
participants and t£e organization, however 4 was significantly greater 
than imagined. The long-term issue of durability and continuing impact 

< , / 



appears* to be a necessary consideration in the original design of 
the Model Evaluation Unit if it, in fact, is* to endure* 

^We have, step by step, developed meaningful products that can 

be used by m^pagement and field personnel,. ,0ur commitment was to 

> 

develop products that could be utilized by, our Agency and other agencies 



oXtr- size,- 



The success of this objective can be measured l>y our Agency's * 
utilization of 6ur developed product's. 



ivT Methodology ' . , \ 

' CONTRACT TASKS • • , ; > . £ • 

' This* section discusses the major areas of ' contract .work as. required 
by the federal contract. ; Each activity, (Task), will be discussed in terms 
of requirements of the' .contract and accomplishment s . 
1. ORGANIZATION , "PLANNING AND' MONITORING 

Requirements '.- This Task required the articulation and refinement of 
a conceptual framework (work plan and 'structure ) for the Program Evaluation *i 
Section's functioning. 'Regular reporting bf contract wprk and scheduled 
meetings with other State Model Units and*feder&l contract officials were 
necessary to fulfill this Task obligation. v ; .» ' • ; 

H Accomplishments - Organization - A graphie descr^pt/on of the Agency 
organization prior to the contract (Appendix..l); .and the Program Evaluation 
Section as structured- for th£ contra ^tii^'sRown in -(Appendix 2). / 
' Appendix 3 displays structure x>? organization as it now exists. The appen- 
dices emphasize the relative position and formal linkages- .of tfre Program 
Evaluation Unit within the .entire* Agency org an iz at ibnal- structure . 

' 'Although the Program Evaluatidn Unit organization was, shown as a ^ 
separately functioning activity within tfo^present Program, Evaluation Unit 
" organization, it did, in practice, function in an integrated manner with 
the existing Program Evaluatipn Unit. The interrelationship was further 
enhanced by the fact that the present -Administrator of Program Evaluation . 
was designated as the Project Manager of the Model Unit and has- direct . 

\ • " ' " ' 

access to\ internal and external linkage§. ^ 

* \ . ' ' i - j 

The ' contract .e*a structure had t d he advantage of giving special emphasis 

' to grant activities while providing centralized management and budget 



control. 
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Planning and Monitoring - This segment of activity has allowed Pennsyl- 
vania to refine and thereby enhance its abilities in organizational aspects. 
We* rapidly f discovered that one can use all types of paradigms and organi- 
sational tables;, however, if there is no re&l communication links the "best 

r * & 

paradigm will fail. As a result, our organizational structure and plan 
(conceptual framework) 'has not changed drastically, but rather has been 
improved in effectiveness. Of utmost importance to the* Program Evaluation 

9 

Section is support of Bureau activities in a timely and reliable manner. 

This is nurtured through interpersonal relationships and constant communi- 

*" > 
.'cation and feedback. The Program Evaluation Section strives to maintain a 

high level of efficiency and effectiveness through the abov£ stated mediums. 

The fulfillment of a working conceptual framework is demonstrated in part • 

1 in the article which is provided in MEU News, flo. 1, June 1981 (Appendix 4), 

"Durability of Units." . / * J 

The* preparation of* the Monthly Progress Reports and Advisory .Committee 

^ " ' • Reports Was timely and ^isef ul in recapping in a continuous way MEU-work 

, f activity. Bi-monthly and "Advisory Committee meet ings .were attended with the 

• bi-mbnthly planning sessions encompassing sharing of ideas'and planning • 

(strategies. These, meet ings revealed the 'diversity of the MEU's and the 

likelihood that a concensus- on many matters may not always be possible. 

-Section meetiAgs on a regular Msis have allowed the moni/toring of our 

lizational* functioif to b^Iccurate and timely,. Interaction between 
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Section staff is at a, high llvel and most cordial thereby allowing adequate 
channels of communication to be sustained. This behavior emanates -to ail 
levels of the Central Office and field re^sultfing in a most satisfaotory f . 
mettiod' to operate . . ' . * 

The Operational Tracking 'System (OTS) for the MEU was developed over a 
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period of months and resulted in increased awareness by staff of MEU resf>on- 
sibilities and facilitated timely planning (Appendix 5). Methods of the # 
OTS were conduced with initial efforts toward constructing cr^fLcal path S 
analysis to form a well coordinated system to track and monitor MEU activities, 

MOB 

Four critical items for planning purposes were developed in the contract 
period: 1) an "Organizational, Functions, and Assignment Handbook" 
(Appendix 6); 2) a utilization plan for Case Service Evaltiators (Appendix 7); 
3) a document describing th£ Program Evaluation Section's goals and objectives. 
(Appendix 8); and 4) a contract budgetary and fiscal system (Appendix 9).' 

Discussion - .The first year resulted in planning and writing the.concep- 

% tual framework. Changes were necessary in years two and three due to external 

K « 

delays of contract requirements and changes in thinking about how the Unit 

J * 

should -function. A now retired^MEU staff member did outstanding work in for- 
mulating organizational strate^ie.'s to assist staff members who provide input 
^Xo the functions of the inter-Un.it and "field contacts we make\ Th£ Pennsyl- 
vania' Program Evaluation Section consists of highly creative aHw flexible 
^ individuals who adjust well to changes in <the« design and Operation of work . 
directed by the conceptual framework.. \ 

Pennsylvania enhanced its organizational skills by, participating 'in the 
contract. , More emphasis was placed on adequate needs sensing \ planning,' 

appropriate implementation and fqllow-up /feedback . The contract required p 

t ^ < * — • 

staff members to work together ^more crosely to accomplish various goals. The 

\ ' • • ~ ' • ' 

result was an mqrease in work productivity, creativity, and^ndividual and 

> 

group satisfaction* ^ • 

Advisory Committee establishment came well after the contract had com- 
menced wor^and initial planning had been "completed. As a result, the impact 
of this group was not as beneficial as if they had been forlfied prior to the 

m 
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letting of* the contract. All. Advisory Committee and Monthly Ffrog<reSs 
. Reports were completed in a timely manner. 

2. TECHNICAL ASSISTANCE ^ . _ * 

Requirements - Basic technical assistance (TA) was to be supplied by 
West Virginia Research and Training Center (RST) , Berkeley -Planning Associ- . 

m ates (BPA), Walker and Associates/National Association of Rehabilitation . 

• * 
Facilit ies (WA/NARF) ., and private consultants. 

RST^as. to offer TA in pretesting of standards/and Facilities Information 
r~ System (discussed later), to define efforts to develop, new evaluation capa- 
city, to describe efforts to disseminate and utilize products, and 'revisions 
of Evaluation plan.- * - • 

BPA was to supply TA in the area of field testing of performance,, pro- 
x cedural*\and project standards (discussed later): 

WA/NARF was to' provide TA in the field testing o£ the Facilities Infor- 
mation System. 

Accomplishments -> The work of BPA and WA/NARF are discussed later in 



this section. 



We have utilized the services of RST for TA. Seyeral Program Evaluation 
♦.Sect ion ^meetings wsre held in late Fiscal Year 1979 to determine how best <to 
, use K£T. As a re'sult, Fiscal Year 1980 resulted in several visits as 'put - 

lihed in Appendix 10. RST visits in Fiscal Year 1981 were made -by: 
'Dr. Nan Brenzel in ^her role as coordinator of various activities; Dr. Dave - 
Molinaro, Training Specialist, assistance in the art of dissemination and 
utilisation; Dr. Don MacLaughin , Research Instructor, as^stance in areas of 
EDP utilization; Dr. Meng-Shu Tseng, Research Associate, assistance in 
statistical analysis; Myra Harris, Graduate Assistant, assistar^e in utili- 
zation of cross tabulations for Standards 6* and 9; and Dave Whipp, Audio 
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Visual Specialist, who prodiided a slide tape on the functions^ of our Section. 
Additionally, Elizabeth Pari of the Institute for Infotmat ion Studies was 
contracted by R6T to traift our MEU in the establishment and use of filing. ' 
system^. 

^cause of heavy contract -related work, we were no£,al,ways able to 

utilize RST appropriately. f However, we .now know^the types of assistance 

they can "provide and desire to utilize thdim more effectively in the coming 

years t^cause of "their specialization in the fiefif of" TrogranT Evaluation. 

Unfortunately the coordinating contract was' awarded well into the 
*- * . , l * 

contract and as a result, Pennsylvania could not benefit completely from 

y$ - ■ ' 

• the sources available through ,RST. 

* • -7 — - 

^ Discussion - This Task required work with RST , Walker and Associates 

> * - ^ >U ' . 

(WA), National Association, of Rehabilitation Facilities~(NARF) , and Berkeley 

I ' 7 

fanning Associates (BPAK Pennsylvania viewed these bodies as definite 

providers of needed information 'and assistance. Pennsylvania approached each 

in a manner that would hopefully lead to a conducive and meaningful working 

^relationship. All of the above were not involved in the first year of 

contract work due -to ^ate awarding of contracts by the Rehabilitation Services 

Administration (RSA). As a result 1 , Pennsylvania was left in a frustrating * 

, situation in the first year with, contract requirements but a> lack of TA 

The first year was spent in plann^ many "what if" situations fin order for 

Pennsylvania to be responsive in the three years allowed to fulfill -the 

* - : . y 

contract. BPA.- Pennsylvania feels the organization did the best they *fcQuld 
under the circumstances. Linda Barrett got the BPA woi*k off to .a very good 
start. J. Mark Rogers succeeded her and tried to' get as much input as 
possible, from the states. Basically, the TA provided was substantial and- 
helpful as well a's the^ effort. WA* - Initial contacts were enthusiastic and 
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optimistic. However, when NARF got the contract for the' field test of . 
Vocational/Medical Facilities Reporting System (Task 5) it was unclear 

i 

to Pennsylvania who would provide TA between WA and NARF. When Pennsyl- 
vania decided not to conduct field tests because of lack of federal OMB 
approval of the forms, ^e received little guidance. Slowly that segment 

of the contract lost its optimism. It appears to Pennsylvania that WA f s 

* - i 

y i 

use as a contractor rather than a sub-grantee may have provided more leader 
ship to accomplish the Task. RST - Because the coordinating contractor 
started functioning only in the second year, Pennsylvania did much of the 
planning in the first year. As a result, second and third year coordination 
by RST was helpful but would have 'been more so if awarded before the first 
year. We received good TA from RST on: computer software training, and 
utilization of Morggntown EDP facility, and the development of a Pennsyl- 
vania slide tape show. f 

37 WORKING RELATIONSHIPS WITH ASSOCIATED ORGANIZATIONS , CONTRACTORS, 
UNIVERSITY RESOURCES AND USE OF CONSULTANTS j 

^ Requirements Establishment of continuing working relationships with 
the following organisations was encouraged: University of Michigan Regional 
Rehabilitation Research, Institute (RRRI), Arkansas Research and Training 
.Center, West Virginia Research and Training Center (RST), Abt associates, 
and other selected consultants to assist in areas cf contract work. 

Ac comp lishthent s - Contact has been established with all of the following 
Arkansas RST; West Virginia RST University of Michigan; Research Utilization 
Laboratory at ICD; Abt Associates; San;Diego State University Walker and 
Associates; Berkeley Planning. Associates ; other MEU states; National Associ- 
ation, of Rehabilitation "Facilities; Pennsylvania Association of Rehabilita- 



tion Facilities; Human Interaction Research Institute (HIRI); Oklahoma 

7 ' 

Rehabilitation ancL Management Program; National Rehabilitation Information 

- * 

» 



Cent*r/(NARIC); Project SHARE; Oklahoma State Clearing House on Training 

17 ' 
Materials; and various contacts with the Rehalftlitation Services Admnni- 

i *. 
stration. Consultants utilized were: Stanley Portny of Stanley Portny 

Associates, Inc. ; John Muthard, University of Flo^da RRI; David Sigman 

* and Paul Games, Pennsylvania State University. These contributors 

are* c%ted throughout this document for their assistance. 

Technical assistance in dissemination and utilization strategies was 

provided by Muthard, Sigman, Portny, NARJC., Oklahoma State, RUL-ICD, and 

HIRI . 

Initial contact was made with Linda Barrett of"*&FA in October of 
1978. Discussion with/BPA indicated that RSA had not yet awarded their 

* A* 

contract . 

' Miss Linda Barrett spoke with the Model Units 1 staff at a bi-monthly 
project meeting hel'd in January 1979, regarding plans for the field testing 
o^the Revised VR Program Evaluation Standards. In March of 1979 a meeting 
was held with Linda Barrett and the PA Evaluation staff, discussing Issues 
and plans for the field testing of the Revised -VR^Program Evaluation Stan- 
dards. * - . ■ 

Initial contact was made with Bruce Maloof representing Abt Associates 
in October 1978.. At that time, it* was learned that Abt Associates planned 
0 to meet with the Model Unites in Spring of 19.79. In June 1979, our Principal 

4 

Investigator met with the Model Units to provide input on the RSA Information 
System by participating in an Invitational Symposium sponsored by 'Abt 
Associates. Our State Agency was present at a RSA Region III Program Evalu- 
ation Forum to hear Charlesf Cole's (Abt Associates) presentation. 
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, Throughout the three years of the contract, contact has been maintained 

with Walker and Associates and/or Natipnal Association of Rehabilitation • 

Facilities " (NARF) . ' ^ , 

The initial contact with Walker and Associates was at the first project 

• meeting in October 1978. A subsequent meeting was held in April 1979., At 

* * this meeting a discussi6n of preliminary plans and issues related to the 
* ■ 

field test of the Facilities Information System was conducted. 

Further contact was limited until January 1980. At that time a meeting 
« 

was held with representatives of NARF (contractor for the, field test), 
Walker and Associates, state rehabilitation facilities, the Pennsylvania 
Association of Rehabilitation Facilities (PARF^ , and the Bureau. At this » 
meeting several, programmatic areas were discussed. These areas included: 
1) number of facilities; 2) types pf facilities;" 3) training; 4) dates of 

9 

field tests, and other topics. » 

Training was conducted on t'wo separate occasions. Once in May and 

* J 0 

once in June 1980. The field test started in July 1980. However, since 

the forms to be used during the field test had not bee^ri approved by 0MB, 

we chose not to participate. 

\ ' Since that time, very little ^contact has been maintained. One subse- 

guent meeting was held in February 1981, At that meeting, the results of 

X 

the field test were discussed. 

Discussion • 

University of Michigan Regional Rehabilitation Research Institute 
(RRRI) - Contact was made and several discussions were held, however, it 
became clear that collection, design of instruments and analysis techniques 
were, in most cases, similar to Pennsylvania 1 s present efforts, 
t Arkansas Research and Training Center - Contact resulted in contracting 

with David Sigman to assist us in. several dissemination -and utilization 



.strategies. ' 
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West Virginia Research and Training Center (RST) - Utilization occurred 
here in several technical areas ^(EDP, filing, disinclination and utilization). 

Abt Associates - Contacts with Abt .made it appear they wanted^ our input, 
however, it was clear that formalized channels were not available for input 
and feedback was almost non-existent. This factor was pro*bably due to 
changing personnel at Abt, 

ICD - A field visit was conducted and discussion of dissemination 'and 

utilization (D£U) strategies, techniques, and operation of their library, 

.... > 
occupied the visit . u 

> . 
San Diego State University - Coordination with the institution occurred 

on the Case Review Schedule Training. * ♦ * 

Walker and Associates, Berkeley Planning Associates, and National Asso- 
ciation of Rehabilitation Facilities - discussed later/ 

"HIRI - Utilization of Thomas Backer in D£U skills, 
_____ % 

NARIC - Continued assistance in the development and enhancement"of our 
vocational rehabilitation library. »^ 0 

Project SHARE, Rockville, Maryland - Site visit to view their D&U 
activities. . ■ - 

Oklahoma State Clearing House in Training Materials - Utilization to 

enhance our vocational rehabilitation library. ^ 
> 

Consultants utilized through the contract were Cinda Barrett, Stanley 

Portny, Paul Games, David Sigman and John Muthard.' We are very pleased with 

the work completed with these individuals. 

# 

Stanley Por.tny - Utilized in areas of D£U, -technical wrijting, assistance 
in final report and executive ^summary formatting, development of conceptual 
-framework and other program evaluation strategies utilized in the. three years 
of the contract. 
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John Mutha rd, University of Florida RRI - Assisted in molding DSU 
strategies . - ~ 

' ( ■ 
Paul Games Pennsylvania State University - Worked with staff on further 

• development of our Profile Analysis$Technique (PAT). 

Davifl Sigman - Assisted in formatting and design of final report appen- 
dices and executiye summaries cover pagefl 

**• . EVALUATION OF FIELD' TESTING OF THE NEW PR OGRAM EVALUATION STANDARDS 

1 5 ■ 

Requirements - Field test the above Standards, completion of independent 
analysis and critical review of Berkeley Planning Associates final. report on 
Standards work. 

^ Accomplishments - The activities of this Task required that Model Units 

work with a Federal Contractor (BPA) who was to be responsible for providing 

< v 
technical direction and coordination in the field testing of the New Program 

Evaluation Standards. Since no federal contract was awarded for- this Task 

during the first year, the primary work by PA MEU for the first year involved 

planning, conceptual analysis, and review of all federal literature ("The 

VR Program Evaluation Standards: 'Final Report", and "Projects Standards \ ' 

for RSA Discretionary Programs: Final Report") and all Berkeley Planning 

Associates (BPA) working papers on the subject. 

^ Several presentations were made by BPA at the bi-monthly meeting to the 
Principal Investigator and Project Manager in anticipation that this firm 
would eventually be awarded the federal contract. Also in anticipation of 
this contract, meetings were held with our Model Unit's staff and Linda 
Barrett of BPA to discuss an approach to the Standards Contract. Pennsyl- 
vania familiarized the firm with its 'current techniques and methodologies 
in approaching the collection of data for the Standards and with our data 
analysis and processing capacity. In this meeting, Miss Barrett discussed " 
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both the Program Evaluation Standards and Project Standards with Pennsyl- 
vania staff presenting a detailed analysis of possible problems in data 
collection and .analysis , and electronic data processing implications of 

;each "of the projected New Standards. Since a Case Review Process .was 
\ 

anticipated as part of the requirements for the New Program Standards, a 
^similar meeting was heldXrith a representative of San Diego State Univer- 

iSity, Jim Tunstall, in anticipation that their Case Review System would be 

1 1 1 

field testing in our State. 

I The Program Performance Standards 1 data collection efforts involved 
, collecting supplementary R-300 type information on a sample of .100 clients 
^recently accepted for BVR services and a separate sample of 502 recently 
(closed clients. The latter group was also mailed a closure survey and the 
I two hundred s^Lxty one 26 closures received a six-month follow-up survey, 
tin addition, a separate sample of one hundred two 26 closures from the first 
Quarter of 1979 was mailed a one-year follow-up survey. The only delay 
experienced by our State in the implementation of these activities was in 
receiving the one-year follow-up survey questionnaire from West Virgipia 
R£T. Our State completed this^ data collection activity by the middle of 
May 1980, submittal deadline. The only data collection activity not com- 
pleted at that time was the six-month follow-up survey which »was due for 
submittal to Berkeley Planning Associates on November 15, 1980. 

The data collection efforts for the Program Procedural Standards 

evolved into a two-phase activity. The first phase required the training . 

— / ' 

and application of the San Diego State Case Review Schedule to 120 closed 

cases in -our State. In addition, the £PA timeliness assessments of the 

same cases were^completed. This phase was completed in February 1980 and 

j 

^ . * 
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the compliance analysis was presented to our Statute in May 1980. Reprocessing 

, . * \ i 

the data to respond tx> th<* Procedural Standards I jdata collection effort 

: * » . I 

required the training and application of BPA's tf-j300 verifications instrument, 
This activity, was. completed (String May 1980, Datja processing preparations 
wers completed and the original source documents Were mailed to BPA. 

, Pre-test assessments h^ve been conducted at every stage of th^4raining\ 
and .data collection effort, to date. In addition to collecting' the data for 
the ^Performance and Procedural Standards and conducting an internal anafysisi 
of the information we gathered, we monitored the expenditure of time across^ 
specified activities rel^t^d to the pre-test. * 

The Project Standards 1 data collection effort involved the identification 
and cooperation of nine projects located in the Model Evaluation Units States. 
Our State had two of the nine discretionary' projects : 1) Skills Training and 
Enhanced Placement and 2) Extended Evaluation training Project for the 
Severely Disabled. A Model Unit person had liaison responsibilities for form 
completion. To date, pur two projects have completed the Project Planning, 
Progress Reporting, and Evaluation Outcome forms, as appropriate. The Project 
Checklist was completed by a member of our Evaluation Unit. Based on pre-test 
assessment telephone interviews conducted by BPA, the instruments were revised 
and reformatted prior to application in the 100-project sample, in which 

Pennsylvania has two more discretionary projects. 

i - 

a ' % 

- 1 On July 31, 1980, "Pennsylvania's Independent Analysis of Data for the 
Proposed Evaluation Standards" Was submitted to our RSA Project Officer and 
Contracting Officer, At the'end.of November 19801, we received BPA's analysis 
of the Standards. At that tinjf^we compared their analysis to our State's 
analysis.. In December 1980, Mark Hogers , from BPA, visited Pennsylvan^f to 
discuss our reaction to BPA's analysis of the Standards. 1 In May 1981, BPA 
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disseminated a "Report on the Pretest of the Revised Vocational Rehabilita- 
. tion Program Standards . 11 We reviewed that report and on June<26, 1981, 
disseminated our- comments, entitled "PA Critical Review of the BPA Program- 
Standards Report" ( Appendix* 11 ) . In addition, we tabulated the results of * J 

- " e 

the Six-Month Follow-Up Survey for the data that was submitted to Berkeley 

Planning Associates on November 15, 1980 (Appendix 12). Our final activity 

in this field testing of the Program Evaluation Standards will he to review 

BPA's training and guidanc^materials, which will be used to facilitate 

implementation of the Standards by users of the Program Evaluation Standards 

\ 

System. 

Discussion - The major activities of field testing the Program Evalu- 
ation Standards and the Project Evaluation Standards centered around the 

„ ■ m 

Performance Evaluation Standards, The players involved in the Performance 
Standards field test for- Pennsylvania included BPA, RST and the Pennsylvania 
MEU. Overall, our State participation in the field test of the Performance 
Evaluation Standards was aiy^exhaustive undertaking. .We gave this field 
testing our best shot and/ hoped -that the Other MEU. 1 s- did the same. Since 
the impact of what comes out of this field test will live with all of us 
fo t r years to come, we didn't want any state agency to', say that Pennsylvania 
was involved in an activity that failed.^ 

<9 

f 

The field test of the Procedural Standards was divided* between the San 

J ■ 

Diego Case Review Instrument ^and their staff and the R-300 Verification 
Review; with, Linda Barrett frym BPA. * 

The v 4 field test of the Project Standards was an activity that didn't take 
much time because we felt that these Project Standards were new and would 
have to gq through many revisions befo're they became law. We just did what 
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what was expected of us. < . _ ' 

If any of these Program Standards can act as^an agent to improve agency 
performance, these field test activities have, been worth" the fiffort 

5. . ADMINISTRATIVE MONITORING ANlT ORGANIZATION OF THE PROG RAM EVALUATION UNIT 

■ ■ ~ <* 

Section meetings on a regular basis have allowed t*he monitoring of our 
organizational function to be accurate and timely. Ihtetacl^fen between . 
section staff is at a high level and mast cordial thereby allowing adequate" 
channels of communication to be sustained. This" behavior emanates to all 
levels of the Central Office and field resulting in a most satisfactory 
method to operate. w \ 

J * - 

The Operational Tracking System (OTS) for the MEU was^4eveloped over a 
period of months and resulted in increased awareness by staff of MEU respon- 
sibilities and facilitated timely planning, see Appendix 5. Methods of the 

i 

OTS were conducted with initial efforts of constructing critical -path analysis 
to form a well coordinated system to track and monitor MEU activities. - 

Three critical areas for planning purposes were developed in the con- * 
tract period: 1) an "Organizational, Functions, and Ass ignmer^' Handbook" , 
see Appendix 6; 2 )*a utilization plan for case service • evaluators , "see 
-Appendix 7; and, 3) a document describing the Program Evaluation Section's goals 
and objectives, see Appendix 8. & ^ 

6. FIELD TESTING OF THE FACILITIES INFORMATION SYSTEM 

Requirement - This Task involved field testing in each of the six Model 
Evaluation States in 1980. Independent analysis o§ data generated and an. * 
actual review of the Walker and Associates/National Association of Rehabili- 
tation Facilities Comparative' Analysis' was to be performed by Pennsylvania* 
jf Accomplishment - The purpose of" this Task was to*<engage iu activities 
"neceSjary to field test the Vocatifcnal/Medical Facilities Reporting System 
designed by Walker and Associates. As in previous Tasks, the activities 
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involved required that the MEU work with a Federal Contractor, who was to 

be responsible for providing ^tecfhnical .direction and coordination in the 

♦ • *- 

field testing of tfhis System. ^ { 

The federal contract for the field test was ? to have been awarded during 

the first year Qf our contract. However, the awarding of the contract did 

< not octeur until the second year-. Because jzf this delay, only conceptual 

analysis planning and review of the appropriate literaliure. was done during 

the first year. > . t ^ * ' > . 

In order to maintain appropriate lines of -responsibility within 

c f o > 

" ^ /> // > «... 

Agency, the details of this work were* Performed by the Facilities and Grants 

» 7/ » 

Cagement Section. This was done un/der the direction of th^ MEU. A 
rough review of the System", and l^ts potential uses, wa*s conducted by the I 
FGM Section and a conjoint analysis *w£s conducted, with the MEU. The results 
of that analysis suggested the following problem areas: 

1. The recording method^ does not separate program ^services if the 

client spends time in more than one area.. For instance, most clients 

< * 

receive vocational evaluation Upon entering a vocational facility 
program. However, t*hey spend most of their time in vocational 

V * * * " * 

training and adjustment. From looking, at the report instructions, 

: , ^ - " • 

it appears that the' shorter program, evaluation, would not be 

'recorded, The review of data over a period of time Vould imply 

that clients. are put into programs without prior evaluation ser- 

vices. This situatioh would also*lead to statistical nonsense when 

the number of clients receiving evaluation services and the diag- 

nostic dollars spefit iru facilities are compared. The number of 

clients would be extremeiyilow compared to the dollars spent. 

2. There needs to b4. soihe space for recording state licensing status, 



s well as accredit citron status. 
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3. There was a general concern that facilities will be hostile to the 
widespread implementation of this report f ormat^unless it is con- 
currently linked^ to a MIS endorsed by the facilities, 'Our particu- 
j lar State has the smallest minority of clients, so we doubt that 
facilities will be willin^to keep the kind of records necessary to 
complete the reporting form. 
^4, From our current experience, few facilities have the data base. *to 
* supply the information. Therefore, reliability measures may need 

to* be added to assure that superflous numbers are not used as data. 
The MEU and FGM Sections met with a representative of Walker and Asso- 
ciates in anticipation that the avard of the contract would be made to their 
company. In this meeting, w$ received °an overview^of the Reporting Sysrtem. 
Our, analysis of possible problem areas in the* collection and analysis of the 
data was also *presepted?l ^ t 

In February 1980,; a pre-planning meeting was held with representatives 
ofaWalker and Associates, NARF (Contractor), PARF, Facilities, and the State 
Agency. The meeting was held to determine the breadth and scope of the 
field test. 

Working with PARF, after the meeting a list of facilities to be used, 
with alternates, was developed/ The facilities selected were queried to 
ascertain their feeliftgs about participating in the field test. Once the 
facilities were selected, training dates and sites were arranged. Training 
was conducted "at two site's: Harrisburg in May 1980* and Pittsburgh in 
June' 1980/ 

The field test began July 1, 1980. However, because .the forms used in 
the field test were never approved by 0MB, we xiecided that it was not in our* 
^est interest to participate. ^ Since we did not participate, PARF handled the 



field test activities for Pennsylvania. 
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Due' to a limited response by the facilities agreed \!^cn^_ihe. Stats 
Agency and PARF, PARF decided to select some additional facilities who were 
TVot on the original list. .This was done despite objections pom our Agency > 

The field test was conducted from July 1 through December 31, 1980. At 
the completion of the field test,^n analysis was prepared and submitted* 
A copy of the analysis is presented in Appendix 13. < 

The main objective of the field test was the development of a system 
designed to meet the information needs of facilities, state agencies, and the 
federal government. However, since we feel that the system developed meets 
the information needs of only the federal government, we are in the process 
of developing our own system. 

At the present time, we are working with the Facilities and Grants Manage- 
ment Section to identify all facilities with whom we have a working agreement. 
At the same time, we are developing a coding system -that will identify each 
facility by type. Eventually each client in a facility will be -identified 
by facility. 

Discussion - The purpose of tfie Facilities Information System (FIS) was 
to provide information to facilities, state agencies, and the federal govern- 
ment, concerning activities of facilities". After reviewing the forms to be 
used and the appropriate literature and after the field test, we concluded 
that the information would benefit only the federal government. # 

The field testing of the FIS was to have begun during the first yearNrf^— 
our contract. However, due to delays on the part of RSA in awarding the V 
•contract, the field test did not start until the -end of the second year 
(July 1980). This created problems for the states in that money budgeted / 
for the first year could not be used. This money was lost to us because" it 
could not be carried over into the second and/or third year of our contract . 
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At first, the quality and quantity of technical assistance was 
very good; however, as. the project continued it began to diminish. Until 
the contract for the field test was awarded to NARF, Walker and 
Associates was keeping us updated as to events taking place and any 
revisions being made. • ^ 

Once the field test was completed, several conclusions were reached 
concerning the FIS. They are: m 

1. The 90-.day follow-up is not applicable to medical facilities. 
This is because once a client leaves the facility, no further 
contact is maintained ♦ 

2. The data collected may not be valid. This is due to the fact 
that facilities with small VR populations do not keep^/the type 
of information required. Also, the staff requirements are 
suchHrhat it does ftot allow for the collection of ;the informa- 



tion. 

3. The information collected does not meet the needs of the facili- 
ties and the State Agency. * 

The coding used on the forms is not consistent with current 
RSA coding. 
DEVELOPMENT OF NEW EVALUATION CAPACITY 



Requirements - New types of capacity consistent with, and built 
on, existing capacity was to be developed. s Continued refinement and enhance- 
ment of Bureau activities was encouraged. 

Introduction -* One of the primary areas emphasized in the initial 
federal request for proposals, "Development of a Model Evaluation/ Manage- 
ment Information Support Unit* !l , N was to "build new evaluation capacities which 
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could be generalized in other states..." and "implement into the State Voca- 
tional Rehabilitation Program new developments in the VR capacity..." , It »is % 
our intention 'to emphasize areas^of basic and, applied research which both * 
meet the specialized needs of this Agency and simultaneously involve the 
application or development of techniques which should be generalizeable to 
many rehabilitation . agencies or settings across the country. " , * , * 
The development of this plan rests upon review of the literature .in 
program evaluation meetings with consultants and recognized experts in the 
field and particularly upon a series of need assessments which have been 
conducted continuously over the past several years but more intensively since 
the advent of this project with Bureau personnel. The fundamental methods 
for these assessments have been intensive contact with key administrators at 
both the State, Regional, and District level in order to determine their 
perceived needs in an ongoing framework. As needs- were recognized at these " 
levels or generated by specialized new programs or administrative require- 
'merits, ^programs have typically been designed^to meet the informational or 
management requirements generated by them. The development of this project • 
and the articulations *of a conceptual framework have allowed this process to 
be enhanced through the statement of a logically constructed system which - v 1 
identified areas in which new needs^ can be anticipated or forecasted before 
they arise" as perceived problem areas. As work has proceeded and- as certain 
specialized problem^ have arisen, it has been possible to specify the needs 
for the Agency's research in evaluation more specifically. 

Particularly germane was the establishment of a coordinating contractor 
and advisory committee which had been projected for the first year of the 
project's operation; however, only started in the second year. Since these 
groups ape federally funded and their role in the original RFP specifies 

\ 



'that they shall have direct input into all evaluation development, their 

impact and direction was late. ^ ^ 

A flexible and open-ended evaluajion capacity was developed; able to 

integrate new knowledge and techniques iftto the current" system so that the 

i 

best possible information is available for management support, policy and 
program development and evaluation. The development of evaluation capacity 
involves two primary emphases: the development of staff capacity and the 
development of program capacity. Such activities are inextricably related 
to the development of the capacity of this staff itself and contingent 

*upon the specific training and -skills which have been acquired by individual 
staff .members. , 

It was determined that certain 'principles shouIQ be emphasized through- 

"but^the project; to guide us in the activities selected for emphasis. ^These 
include: ■ 

1. Agency ^needs are related to evaluation objectives. 

« 

2. Quality is assured in evaluation data. 

3*** New evaluation program areas will be developed/ , 

4. Evaluation programs will have nationwide implication wit»h a 
potential for use by all states. 

5. Evaluation' programs will-be consistent with and built upon' * 

* * 

.existdftg evaluation capacity. 

6. v Evaluation programs" will be flexible to allow for 1 changes in 

, laws, policy, and continuous 'updating of the evaluation programs^ 

a 

% • In reviewing those areas originally projected fpr development, it was 
found that the studies suggested met the principles outlined and we intend 
to continue* as originally projected. Thes^ studies are shown schematically 
and broken down into the development of increased capacity of present evalu- 
ation programs (Appendix*! 1 *) and the development of new evaluation capacity 



(Appendix 15). In the following pages, £ brief overview of each study is 
given and, in certain instances, supporting detail or methodology is pro- 
vided in attached Appendices as noted. 

Discussion ^ Great satisfaction was derived from working on this Task. 
The development and refinement of the products in this Task were completed 
in a well -planned manner. Before any activity was undertaken, needs sensing 
was conducted. This thoughtful first step was a direct result of dissemin- 
ation and utilization training received by several staff members from 
Thomas Backer, early in the contract. 

As a result, the activities completed w£re generally welcomed by admini- 
strators ,. Central Office, and field staff. In particular, we are pleased 

with the expansion of * our Similar Benefits Pilot Project and the Case Review 

r 

Process. Streamlining the "request for information procedure" was viewed as 
very positive also. Another extremely successful activity was development 
*of the Technical Assistance Center. which is widely "utilized. 

Intangibles are the skills and training acquired by the staff as they 

V 

functioned in duties necessary for success. Contacts with consultants and 
other professionals presented new stimuli and created a real learning environ 
ment for staff members. ^ 
a. SECTIONAL ORGANIZATION 

This Task was planned and developed through the three years of the con- 
tract. Initially placement of the Program Evaluation Section, on an organi- 
zational chart, appeared to be very important. However, we found that con- 
tract work enhanced our ability to communicate in an effective manner with 
all levels of the Bureau. Contract work helped us realize that interpersonal 
dynamics were more important th&n physical location of a unit or section to 
the Director. Our final functioning as a Section culminated in the "Program 
Evaluation Section - BVR - Organization, FunctioA^, and Assignment Handbook", 
which has sections included in Appendix 6. 
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b. DEFINED TERMS PROJECT . 

ft 

♦As the contract work activity progressed, it became apparent that con- 
solidation of all terms used in day-to-day'MEU functioning would be helpful* _ 
As a result, a now retired MEU staff member Shouldered the responsibility of 
organizing terms used, defining them, using group input, and" developing the 
final product as a "Glossary of Evaluation Terms", which is included in the 
"Program Evaluation Section - BVR - Organization, Functions/ and Assignment 
Haridbook" {See Appendix 6). 

c. TECHNICAL ASSISTANCE CENTER * 

The Program Evaluation Technical Assistance Center was established as 

i 

the result of Pennsylvania BVR receiving this 'federal contract which mandated 
that a library dealing with program evaluation be developed. ^However, as the 
work progressed it became apparent that there was a need for a more extensive 
library which would serve our entire Bureau. As a result, the l^rogram Evalu-v 
ation Section Library has approximately 2,100 holdings covering many areas 

4 

of vocational rehabilitation including hew rehabilitation trends, current 

legislation, specific disabilities , assistive devices, etc. Since its inCep- 

tion in January 1979, the library has disseminated over *1, 600 articles and 

books to the staff of the Pennsylvania BVR for -their use. It has proven to 

be a useful soruce of information for our staff and a means by which they 

» 

can obtain ^current information in the field of rehabilitation. Appendix 16 
gives details on the methodology used'to eatablish the library, the services 
it provides and the means by which material is disseminated to the users of 
th& library. 4+ 
d. - STAFF TRAINING^ ' N 

Staff training of ifhe MEU was conducted in' a variety of ways. During 

the first two years emphasis was on individual afid group training.* A needs 

c \ * • 

assessment of; training for staff was conducted by West Virginia Research and 

* * ' ** * 

Training Center; however, its dissemination by R&T w^**rfot timely. As a 
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refeult, Pennsylvania, basically, for the first two years guided its own 
training and the third and final year resulted in some utilization of the 
R6T training needs, assessment . The?e delays can be tracked to the late 
awarding of the coordinating contract • 

Courses taken by members of the unit were: ^ 

1. "Easyt-rieve" for retrieval of information 4 

(7 

* , « * 

2. "Quantitative Methods ift Statistics" 

3. .S'StatisticalWalysis System 11 

• ."Introdyction to Public Admin^trationV- \ 

5, "Proposal anH Grant Writing" 

6, Several MBO Programs s 

7, "Similar Benefits" conducted by National Paralegal Institute 
8;- National Institute of Justice Workshop on Program Evaluation 
Several types o^ group training presentations were arranged: 

1. "Technical Writing" - • - 

2. "Program Evaluation Filing" 

3. "The San- Diego Case Review Schedule" • 
4 f Time Management for Secretaries"^ 

5, . "Facilities Information System" 

Additionally, the Region III Program Evaluation Forum allowed individuals 
to gain a variety of training experiences in the twice-a-year sessions . 

Pennsylvania was eager to utilize training ^available, however, in many" 



cases geography and contract-related work schedules did allow us to* use 
training funds in 'larger manner. 

A Pennsylvania MEU staff 'member also maintained" a listin'g of current v 
training available to members. ^The listing was circulated and updated on a 
regular basis,* see Appendix 17 « » 
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e. ' CLIENT AS$ISTANpE PROJECT EVALUATION PLAN DEVELOPED 
The Program Evaluation Section developed the Pennsylvania Client 
Assistance Project (CAP) evaluation plan. This plan was developed with 
assistance from the Facilities and Grants Management Section, field offices 
and CAP staff (See Appendix 18). 

The Pennsylvania MEU submitted regular reports on new evaluation capa- 
city in a timely fashion. These reports reflected the progress of our 
developing and refining of products. The Appendices for this section-reflect 
the development and refinement of Pennsylvania Program Evaluation Section 
capacity work. 



f . REVIEW AND IMPROVEMENT OF PENNSYLVANIA BVR MANAGEMENT INFORMATION 
SYSTEM - * 

During the three years of 'the contract ,' considerable effort was focused 
on improvement of j:he Bureau's Management Information System. To this end, 
several new/revAed Reports and procedures ,for requesting information were 
developed. 

An indepth discussion of. this subject can 'be found in Section VII. The 

new procedures, while being developed because of our contract, have become. 

<an internal-part of -our .Sections operating procedures apfi will continue 

» 

after the end of the contract. 

g. FOLLOW-UP SURVEYS OF VOCATIONAL REHABILITATION CLIENTS 

The general purposes of 'the surveys are to establish criteria for evalu- 
ation program effectiveness and for increasing program accountability. "The 

.evaluation object iyee were to insure tfiat clients closed rehabilitated 

i 

retain the benefits obtained from the rehabilitation process and to insure * 

that the client is satisfied with vocational rehabilitation services as 

\ 

developed with the counselor. See Appendix 19* " * , 



The methodology used to see if the clients retained benefits was a five 
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percent stratified random sample from each district office's Status 26 , 

• * 

closures. A questionnaire was developed and 'mailed to the selected cl'ients 

< 

along with&a business reply envelope. Three weeks later a follow-up post- 
card was mailed to those clients who did not return the questionnaire, and , 
three weeks after the mailing of the postcard- a follow-up letter, a 
questionnaire and business reply envelope were mailed. When a questionnaire 
is returned, a thank you letter is sent to the person and if the. person 
requests further services, his questionnaire is forwarded to the district 
office in his area. The above methodology is used for. our client satis- 
faction survey except for the fact that we not only sample rehabilitated 

J 

* -c 

cases but also non -rehabilitated cases (Statuses 28 and 30). 

The results of previous surveys* indicate that all clients wliose cases 
were closed Status 26, rehabilitated, received some rehabilitation services 
and may, have been immeasurably aided, even though circumstances existed 
which prevented the attainment of a suitable occupation for a period of one 
year. The findings showed that 85* percent of the clients closed Status 26, 
rehabilitated, in Fiscal Year 1979, are presently employed. Another goal of 
BVR is to insure that successfully closed cases represent a wage gain and a 
wage achievement sufficient for a reductiqp in economic dependency. Basically, 
it determines ^if job earnings aJdow a client to be free from other forms of 
financial support. Since the average earnings at time of follow-up were 
$160.83 we feel that we have achieved the above goal to some degree. 

In our client satisfaction survey, the overall clients' feelings of the 
services VR provided them showed satisfaction of 85 percent acros£ the State. 
. Two major areas showed satisfaction under 75 percent, statewide: 1) the 
results, of medical services received 'by our clients showed satisfaction of 

\ \ * 4 

74 percent and 2) the benefits from training received by our clients showed^ . 

. ~ 

satisfaction of 65 percent. This could. "indicate that the quality of services 
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provided by our suppliers or vendors to our clients is not the best quality. 

<* * 
A third area that needed revision was the helpfulness of a counselor in 

providing placement services . 

The conclusion that can be reacty&d' is that BVR counselors should con- 

sider the market availability and a client's potential when writing an 

IWRP so the training isn't provided for an occupation that is either 

obsolete or filled to its capacity. Another area of reform is utilization 

of vendors by counselors. 

, , To date, the Bureau of Vocational Rehabilitation ' is currently developing 

a Job Bank placement systern to help placement counselors ^in the district 

\ 

offices place severely handicapped clients. In the area of quality of 
services by vendor, we are currently evaluating workshops and rehabilitate 
centers, business schools, and other various training institutions. 

The information from former clients is t evaluated tol determine utiliza- 
tion by our Bureau for changes that might result in more effective services 

- A * * * * 

to clients. Recentlyjybutf survey showed that many client^ felt, that they 
needed more plac^me^tj services. As a result of past survey's, the placement*, 
program was strengthened to meet the Alien.t' need. * 

For Fiscal Year 1979 annual surveys we will cross-vklidate information 

v 4 

4 „ 

* V 

'on the questionnaire as submitted by the client^ to information on our cotn- 0 * 

r * * 

puter file!. An example of 0 this cross-validation would be 'if a previous 

' / » 

client Stated he received a particular service as stated on the questionnaire 

% * , 

we would then check to see if the f counselor noted this seryice oh c5ur com- 
puter file and vice versa. • , 

The other area\f expansion will be the cross-tabulation 'of JLtems on- 
our computer file to specific questions on t;he follow-up. ^surveys (-See** 
Appendix 20). We have completed these cross-tabulations of item*s # ^using a 
software package, "Statistical Analysis Systems" (SAS) and*the t&£Hnical 



assistance of Wyra Harris from the West Virginia Research and Training '< 
Center at M org ant own, West Virginia. 

fiase^upon the fboye study and the field testing of the New Program 
Evaluation Standards, our State Agency has develpped a new follow-up 

JL 

survey that <we began using in March 1981 (See Appendix 21). ^ 
*h. SIMILAR BENEFITS w 
An area which was investigated by the Program Evaluation Unit (MEU) 
of the Pennsylvania Bureau of Vocational Rehabilitat ion (BVR) was the area 
of similar benefits (those client rehabilitative services which can be 
obtained from sources other than the Vocational Rehabilitation (VR) 
Agency).. VR has traditionally relied on the resources of other agencies to 

assist in the rehabilitation of the disabled, but it was the -Rehabilitation 

\ ' " ' 

Act of 1973 which emphasized the need for VR agencies to be more accountable 
in terms of identifying and utilizing ^imilar benefits, and listed specific 
services to be sought from other agencies before being provided by VR. ' As 
a result, similar benefits has become increasingly important to VR agencies. 

• * . . ■ . - ... , 

The p~urpose of similar benefits is, of course, to maximize the total 
amount of rehabilitation services available to' handicapped individuals by , 
encouraging and assisting VR clients to seek and obtain other services to which 

% - . # 

tihey are entitled before utilizing VR services. VR tries to match the complex , 
and individualized needs of each qlient with an equally complex array of special- 
ized services. Thus, appropriate services are mobilized for the client and the 
best* use of community services achieved. A by-product -of this activity is the 
development of mutual understanding and respect among service providers. 

The possible ^Mnef its of similar benefit utilization are: 

* • * 
District Office Level - \ 

1. « Increase counselor awareness of availability of similar benefits and 

the. need to scrutinize each for the client. 



2. Identify the number of clients who utilize similar benefits! 

3. Identify the category of services, in which /similar benefits were 

s 

used. • 

4. Accentuate the need for counselors to maintain effective liaison 

with agencies and providers* — * 

I 

5. Allow case service monies to* be utilized to an e^en greater degree. 

K I V 

6. Decrease th£ return of case service monies previously authorized. 

7. Encourage district offices to seek out, obtain, and utilize similar 

f ' ■ 

benefit providers on an ongoing basis. / 

8. Encourage better coordination* of district off ice "ke£ivities and 

* communications through the development and maintenance of a similar 
I 

benefits directory. 
Central Office Level 

1. r Determine the impact on case service funds of similar benefits on 

a district > regional, and statewide basis. a 

2. Identification of the number of client $ases and types of "services 
. that have utilized similar benefits. 

3. Assist in identifying areas* in procedures that need clarification 
or modification by field staff with prior action by Training, ^ 

policies and Procedures, or Case Service Sections* 
Development of expertise in monitoring ^nd eventual refinement of 
~ similar benefits procedures. „ 4 * 

5. Build a data base to be used by Case Evaluators to review similar * 
benefits cases. . 1 

6. Provide greater emphasis for continuance of liaison work in cooper- 



ative agreements. * . 4 

Collecting and assembling the^data as proof of similar benefit 
utilization that can be used as justification .of accountability to* 

' i ••• ' * 

thejstate government and external agencies. 
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The need to make use of and document similar benefits has always* been 
acknowledged by BVR. For example, BVR. has long workea with mental health 
and mental retardation agencieVto obtain services for clients, and .BVR 
regulations require that state, grants, Basic Educational Opportunity Grants 
(BEOG) , and other sources of training costs be considered/and' utilized for 
eligible clients. The client ^ase. files contained documentation .of similar 
benefits usage from the better-known sources (e.g., BEOG) but there has 9 
never been a uniform- definition of similar benefits, a directory~of available 
similar benefits for use by counselors, nor any collection -of similar benefits 
data. Thus there was no way of proving «r disproving, s^lar benefits usage. 
It became increasingly evident that there was a need "f ojTfonar process^ to 
record, collect, analyze, evaluate, and disseminate 'similar benefits infor- 
nation if theW was to achieve its ultimate goal for similar benefits, that 
of tneir oplpnum use. Soothe Pennsylvania MEU undertook the task of developing 
iul* a formalized structure in the hope that it would assist in establishing ' 
^£ as Vline.,for fujture work-in. this area. , . ,* ' . 

. The ^an ^k as finallyfcesigned has the following major' objectives: 
'i.; definition of' similar 1 Benefits ' ' . ' '\ 



2. - Implementation ,of° a.stat'^e Similar benefits system 

3. Collection of datajpp srfmU^^flfs via the teleprocessing units 
^ Thxs tjask. involve two„ phases. The first phase 0 is to collect data 

' when' the case is .closed, via the; teleprocessing unit on the following: 
A v Identify clients 'cases that use similar benefits 
at B. Identify Services in whi6h similar benefits are used* ) 
The second phase is a pilot study in one district office that will 
collect the following dat^by the teleprocessing unit when similar 
benefits are being prov4ded: * 
- A/- Identify clients' cases in which similar benefits are used * 
-B. Identify th< sources of similar benefits utilized - 
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C, Identify the services in which similar benefits are used 

D, Identify cost benefits resulting from similar benefits utili- 
t zation ^ 

Analysis of t^ie data collected 

•A complete review of our similar benefits activity is included in 

, Appendix 22 , ' • r> 

i. PROFILE ANALYSIS TECHNIQUE - H * 

This is a technique whicfh features a variety of techniques to conduct 

appraisals of the functioning of the vocational rehabilitation process. 

The statistical and psychological methods of expressing normal distri- 

but ion are jointly chaired by Ijpth psycholpgy and statistical textbooks^ 

> 

The normal curve distribution is v a method of comparing data through the use 
of .standard scores so that valid comparisons can be'made ,of data that follow 
-the normal curve distribution parameters , - 

Historically, the normal curve distribution and standard curves have 

r « 

been used by educators to grade and make comparisons of student achievement, 
Tne* military also used these methods through the use of stanines in World 
War II to test and measure military ^personnel achievement. 

The Pennsylvania General Agency of the Bureau of Vocational Rehabilita- 
tion'has chosen this method of making comparisons^ to meet several needs of 
the "Bureau^ s program of serving handicapped clients, „ , * 

First , # we were searching for a method to equate the rehabilitation per- 
formance of our district offices. Sedbnd, we were .looking for a measure to 
evaluate our performance of our district offices. Third, *we were looking for ^ 

a measure to evaluate 'our performance with other states and 'the nation (as 

* * ° 

required by the nine Federal Evaluation Standards), 4 Fourth, we were looking 

for a method to show the atrengths and weaknesses ,pf olir program. Fifth, we were 
searching for a method and procedure that would increase our evaluation capacity 

and effectiveness. Sixth, we would *£hen utilize tfteir increased knowledge to 



make recommendations for program improvements and accomplishments. 

Profile 'Analysis 'Technique shows on a visual profile where\a state, 
region, or district ranks in program accomplishments. 

Profile Analysis Technique, in the opinion of the author, has the same " 
potential for program evaluation as the original census data tabulating 
machine had for today's computers. It. is a technique that is in it's infancy 
but has the potential to be used in any effective research for any type of 
program. If the terms^are defined and the proper research procedures are 
followed, the Profile Analysis Technique can <be utilized productively by 

L ' 1 

any corporation, government, or person. ' 'V 

Profile Analysis Technique has been greatly expanded under developing^ 
new evaluation capacity of our federal contract. 

A m odel profi le is .currently being developed, wTiich will update our ^ 
current output variable profile and develop two new profiles. A process 
profile is being developed which examines the impact that selected process 
^variables have in |he success or failure of a case. A referral profile is 
also bging developed, which wijll expand our knowledge of which referral source 
variables are most productive. Consultation .work on" predictor factors is 
scheduled with Pennsylvania State University. 

The three profiles, when developed, wiJJ be combined into a sirtgle evalu- 
ation profile of Bureau progress. 

The attached appendices give you both the write-ups and the graphic 
tables" of the past, present, and proposed future profiles for the Pennsyl- 
vania General Agency (See Appendix 23). 
j. CASE REVIEW PROCESS • 

We in Pennsylvania feel that in developing a comprehensive program, evalu- 
ation plan a Case Review Process should be one program methodology in evalu- 
ation activities. A Case Review Process has the capability of 'identifying * * . 



base line case service patterns, ftnereby generating information for the 

. € ' J 

initiation or improvement of decision making in policy j^d. procedure. 

The extent' and degree of thjb structure and activities of the Case 
Review Process is contingent upon the personnel allocated by the -state to 
evaluation and the needs of the state as seen by top management and other 
inputs. j 

Because of ' the proposed Federal Evaluation Standards, which are currently 
being field Rested by the Model Evaluation Unit, each state in the c6untry 
will probably have to develop its own Case Review Process or utilize an 
existing process developed by someone else. 

•- - i 

the Pennsylvania Case Review Process had been in existence prior to 
tjie contract. However, the contract offeredAus the opportunity to refine and 
further develop this type of program evaluation capacity. 

? A complete, description of how the Case Review Process ivolved is.JLn- 
ciuded in Appendix 24, / 

k. DISSEMINATION *AND UTILIZATION 

Requirements - Th|s activity ^is divided }nto internal and external 
segment s. Internal requirements revolve around development of several 

mettibdologies to increase Communication of the Section. These include a 

\ - j 

Technical Assistance Centek distribution system, consultation, training,' 

i v 

and expansion of dissemination and utilization techniques. The external plan 
includes joint acHyitie^with other MEU's or the coordinating contractor. 
These activities include training, consultation, development of users list, 
and promotional activities, . 

Accomplishments - This Task has benefitted from technical assistance 
provi'ded by a varied group of experts. ' As a result., the art of dissemination 
and measuring actual utilization has received heightened awareness in the 

v • ■ 

Program Evaluation S^ot ion's activities 



I. 

An internal dissemination and utilization (DSU) plan was developed, 
early in the contract to guide our efforts (See Appendix 25). 

We adopted and modified the Backer Mpdel, Appendix 26, utilizing needs 
sensing as a first step in any type of process. The selection of -material, 
to selected audiences was also an early activity in our variety of activities. 

A computerized list of users of program evaluation was mobilized early 
in the contract with the assistance of Arkansas RST. As a result, break 
outs of different segments of users were identified for future utilization. 

Several innovative vehicles are now utilized within the Program Evalu- 
ation Section for internal DSU including: 

1. Preparation of a brochure' describing P£ functions to be disseminated 
tcf 3ureau employees (See Appendix 2.7). 

2. + A bi-weekly newsletter disseminated to all PE staff (See Appendix 28) 
. 3. Operation of the Technical Assistance Center (See Appendix 16). 

4: Distribution of the "Rehab Brief" with insert (See Appendix 29). 

Additionally, presentations were made at regional, district, and state- 
wide meetings by MEU staff and articles. appeared in the Bureau's official v 

» • • «" 

publication entitled "Success."* / 

■ / ' " ' h 

' External - A fine external DSU plan was developed through collaboration 



of "the MtU's and .West Virginia Research and Training Center, the coordinating^ 

^ * m <m 

contractor. However, the dissemination of MEU work was not as effective as 
it could have been since the coordinating contract was not awarded uirfcil the 
second year of the contract." . \ / # . ~~\ 

Pennsylvania MEU participated in the preparation of the "MEU Newsletter" 
and "Manuscript of Special Observed Activity" articles/ - Jhe titles of tfce 
articles are: "Case Review Process" by Harry Guise; "Eva^tfj^tion Section 
Library" by Leah Kuhns; 'Jffhe Model Evaluation Project in ^A" by Jenkins; 
"A. Project Manager's, View" by Harry Guise; "The Durability of a Model 



Evaluation Unit" By Don Hossler;^ "Grants-Contracts , Budgetarily "Speaking" 
by Harold Seit-zer'; Experiences and Impressions of a Case Service *Evaluator" 

by Paul Saupp; and ' "Secretary Reflects on MEU in PA" by Zelda Peters (See 

t " > ** 
Appendix 30>) ' ' * * - 

f— - i 

Additionally, gre,at detail was given to planning for the National 

4 > s * . : 

Conference held Jun$^30 -.July 2, 1981 (See Appendix 31), 

■ f ' • >i • > 

Pennsylvania also effectively participated in the Region III Program . 
* Evaluation Forums to collect and disseminate information. 

Planning for more than a year culminated in a Pennsylvania Sister 
•States Conference held September 16-17, 1981 (See Appendix 32), 

A, special .cover was designed for all internal and external DSU packages 
during the contract period (See Appendix *33 ) . 

/ 

Discussion - Dissemination and utilization activities for the three-year 



contract were divided into two segments - intei^nal and external. Several 
staff members Stained training in the strategies . involved. Once again 
Tom 'Backer's plan, alluded to earlier irv this document, was utilized in all' 
DSU activities. * 

Paternal - This activity was extremely satisfying in terms of techniques 

5 1 k , . • _ 



learned. an N d utilized by staff members. Techniques used in the "Rghab Brief" 
and "Evaluation Update" are notewcfrthy. 1 Additionally , the Task 6 products 

-4r 



listed earlier are evidence of the utilisation of a basic DSU strategy. 

External - A problem developed because of the late award of a coordin- 
ating contract,. As a result, Pennsylvania began developing a DSU list of 

\ possible users for dissemination purposes during and at the end of the con- 

•* < $ 

tract . The entire first year, in terms of DSU, suffered because' of a lack _ 

' * ** 

of direction. Areas developed by RSTJ such as MSOA and MEU Newsletter 
(mentioned earlier) were beneficial in helping DSU k "catcl)/up" wit^i contract 
activity. • * 
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Overall, D£U worked well for Pennsylvania as it scrambled to utilize 
appropriate consultants and "dig" for informati6n on proper application on 
such techniques. 




V, PROJECT EVALUATION 
The ^importance of goal setting fs important in any system 
in order to facilitate appropriate program evaluation.' Measure- 
ment of achievement of project objective^ is critical to a fair 
appjaisal of what has occurred. 

As a result, Pennsylvania fcas utilized consultant Stanley 
. 'Portny to complete an independent evaluation of our three years 
.'of contract activity. Mr. Portny relied on our Monthly Progress 
.Reports, Advisory Reports,, deliverables, annual reports and our 
responses to the federal contract 'proposal. 

The "following is the evaluation based on Berkeley Planning - 
Associates'' Project Standards fox RSA discretionary programs. 
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PROJECT REPORTING: PLANNING 



Title of Project: 
Address : 



A Comprehensive State VR Pm**. 

^^^^ «"o«g h 



a Model 



Telephone: 7 17-787-73 12 



Funding Agency:- - Rehabil itation Service* ah * ■ 

servi ces Administration 

Address: Washington, DC 



Telephone: " 



* * 
Grant Number: HEW- 1 05-78-4009 



Monitoring Officer f Dr. James E.' Taylor • T , \ 

1 Telephone: 



Nana and Position of Person Completing this Form: * ^tanley^Po^ 

Consul t ati t 



Name and Position, of Person Reviewing this Form: * 



D^te: 



Date: 




J* 



STD. 
NO. 



LINE NO. 

1.10 
l.U 



PROJECT REPORTING FORM: PUNNING 



PROBUM INFORMATION 
v 011 Prior related study:. 



None directly related 



Abbreviated study findings f_ 



Project: 



BVR Model Evaluation Unit 



=F 



r 



1-10 

1.11 



02: Prior related study 



Abbreviated study findings: 



TARGET POPULATION INFORMATION 

. Primary Target Group(s) : 
1.20 Type 
All 




Disability Code No. 

aii : 



Target Population Site : * 
s 1.22 National: • 



'A V 



1.25 Local: 



ERIC 



(ATTACH ADDITIONAL PAGUS AS f Nl;l:DCn) 
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PRaiUCT REPORTING FORM: PLANNING, continued 



STD. 
•♦0. 



G:) 




LINE NO. 



1.24 



1.30 



.10 



2.20 



2.21 



TARGET POinJUTlON INFORMATION , continued 
Sourcc(>) of above Jrit'onaJtion: 



Additional luforwtion or events: R ^ A has been -sponsoring 4 program of national VR evaluation activities for almos t 
ten years,* Experience ha d shown that, in addition to developing information for the national program offic e , 
it was highly desirable to create a mechanism for translating this information into format s relevant to and 

usable bv state agencies and to help sta t e agencies' to e stablish procedur es for using the information in 

pxpgram operations « — . ' 

6* 



LE GISLA T IVE INFORMATION 

P.L. 93-il2, as "amended by P.L. 93-516 
and P.L. 95-602 " • r 

He levant Acts ~* 



TAX 



Relevant Regulations 



project .i^ion with respect "to legislative intent: To develop a model for a comprehe nsive program and policy sygtem in a State 
VR agency which. effectively uses evaluation information; to develop a viable mechanism for field testing 
nr ftp n fl?< j nfltjpri ft* evaluation and management syst e ms and to transfer information about the development ^ a nd 
implementatioa of this model to other state VR agencies* y 



Potential impact on rsa nroun.m: Result* of this projec t will enable Pa* EVR t o allocate scarce resources more — 
efficiently and,- thereby-,, to pro v id e hig ^-quality^-aervicec to gr ea t er— numb er s of p e ople j to aooioC in th e 
manageme nt of caseloads and the.makir fg of service d e l i very dec is i on ft^Jthe jehy improving the overall qu a li 
of services; to. enable more .comprehensive and responsive program^planning and., thereby, to enable more 
acc urate assessments " ' £ ' " " + # 



more ac cur ate- and. 
federal prpgram object 




6. 
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I'MJECT REPORTING FORM: PLANNING, continued 



COALS ANO OBJECTIVES 

3.10 G0AIS 



01; 



To develop a model of a 
comprehensive program and 
policy system wtiigh effectively 
uses evaluation information. 



- 3 t 20 OBJECTIVES ' 
. (see attached 'pafl ea) 



3.2! MEASURES 
( see 



DATE(S) TO 
3.22 Bf ACHIEVED 



attadied^pages) Not included sinrp this 
evaluation is being pre 
pared. at the completion 
» of the project. 



02: 



^J^alS^ (see attached E gges) 
VR standards and^o, evaluated " K2SS£i- 

their effectiveness ' for VR 



state agency management 



jsee_att ached pages) Not included sinr* this- 
4 evaluation is being pre- 
pared at the completion 
of the project 



03: 



\ Td field test the Faciliti es in^rr.^ 

reporting System and t5 

evaluate its effectiveas for 
VR state agency management. 



|es) . 



,(see attachedjageg) Not included sinc e this 
. evaluation is being pre- 
pared at the completion 
£ o^the project. 



04: 



g4^ d and **™l o? new (see attached pages) 
evaluation capacities ~ page s; 



(see attache^pages) Not included' sW this 
evaiuatiqn is being'pre- 
-pared at the completion 
of the project. 



OS: Tp develop^ internal 

(within the state VR — 
agency) communication, 
dissemination and 
utilization networks 



(see attached -page s) 



ia^l^tached^^ No^include^ 

evaluation is being pre- 
, , v pared at the completion 
of the project 



06: 



To devel op external - ( flee attahh ^ A \ v 

commun^cation, dissemination * 
,an$i utilization network*. J 

c 4 ( a ™:H ADOITfONAt, PAWiS AS NIU.DRD) 



-i^e^tta^p^ 

evaluation is being pre- 
, " _ pared at the completion 
,of the project. 



STD. 
NO. 



, I'ROJECT REPORTING FORM: PLAWNIHC, ^continued 



LIKE NO. 



COALS AND OBJECTIVES 

3.10 COALS 



07* 
I 



To develop plang f£r* 
contijiuLng the .Model * 
Unit* after the termination 
of the thr^e-year contract 
periodr , 



3.20 OBJECT W:S 
(see a ttached page«^ 



3.21 MEASURES 



DATE ("S) TO 
3.22 B r ACHIEVED 



(aee . arfflfhpd pagpfQ Nor itw»1hh£i «n nrf > this 
evaluation is being pre- 
^ pared at* the completion 

of the'project 
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$ (ArrACII ADDITIONAL PAGES AS NM'.DFD) 

- I 



« 4 

Goal Q ' 1; To develop a model 6f a comprehensive program* and policy system 
which effectively uses evaluation information. ' 

Objectives : * 

■ c * . 

I. To develop a strong evaluation unit that is able* to assess 
management information needs and provide information to 
management* - 

¥ - Measures: See Goal 01, Objective' 3; Goal 04, Objectives 

- 1,2,3,4; Goal 05, .Objective 2. * 5 

. 2.. To insure that the eval^tion unit is in an organizational 

position which affords, it ready access to the Director and 
other top agency administrators. 

Measure: Organizational location of the MEU • 

3. To employ in the MEU personnel who are £ble to: (a) con-' 
ceptualize agency information needs, plan* and measurable 
. • objectives and use" them in evaluation studies; (b) conduct 
evaluation surveys and other studies; (c) work with data 
processing; (dX analyze data; (e) create evaluation .reports 
which are clearly written, easily understandable and of 
potential utility in agency decision making; and '(f) work 
with agency personnel to assure the development and dis- 
similation of evaluation information. 

- Measures: 1/ Number of MEU staff. 

2. Staff position descriptions. 

- , t 3. Staff capabilities. 

To conduct special studies* and projects of interest to Pa, 
BVR, in addition to those specifically required for federal 
reporting purposes. 

Measures: 1. Number of special projects and. studies c on- 
ducted for Pa.- BVR during the' past three years. 

* 2. Number of routine reports periodically, 

generated for Pa. BVR. 

To develop a -conceptual framework for program ^valuation 
within the VR agency. ' * ' 

Measures: I . Submission of preliminary" conceptual framework. 
2. Submission of revised conceptual framework. 
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Goal 02 : To field test the Revised VR Standards and to evaluate their 
effectiveness for VR state agency management*. \ 

Objectives : r . 

1. To complete plans for field testing the New VR 15 Standards 
in Pa. BVR. 

* ' i 

'J . Measures: ,1. Establishment o£ contact and working 

r- . relationship \^ith the federal, contractor 

V * (Berkeley Planning Associates) ' 

' 2. Completion of preliminary activities required 
^ * for the field test. 

v^^ , ' - ' fc Participation in training required for the 

. 1 ' ft - field test. ' . 

v ^4. Preparation of a- report detailing Pa. BVR 

\ * ^ plans for tf?e field tiest. 1 

i . * 

2. To conduct the field test of the Revised VRjStandards . 
- ~" V 

* Measures: 1. Conduct of closure survey. 

* 

2. Conduct of follow-up Surveys. 

s ^ 3. Accumulation of required program data.. 

4. Analysis of the results of the field, test. 

' 5. Collection of data for the New^VR Project 
* Standards from pro jects ^selected to participate 
in the field test. M » \ 

. JT~ 3T~'~'To~px^are~~reportsr-on-~the results of the field test. 

Measures: 1. Preparation of a 'report on the results of the 



Pa. ElVR field test of the VR program and project^ 
« - standards. 

> 2,. Preparation of an independent 'analysi^ of the 

Revised VR Standards, based upon BPAfreports , 
describing their assessment of the f&eld* test. 

% • f , 

Goal 03 : To field test the, Facilities Reporting System and to evaluate its 
effectiveness for VR state agency management.* f & 

Objectives: « " 

1. To complete plans for field testing the FacilLta.es Reporting 
System. 0 • * y 



? erJc 




^1 



Measures: 1.- Establishment of contact and working - 
. , relationships with the federal qontractojr, . ^ 

NARF and Walker - and Associates. - 

2. 1 * Completi6n of preliminary" activities required 
. • for the field test. ' * 

3*. Participation in training required for the 
field test. 

n r • ■ ■• v 

2. To coordinate the field-test of ■ the FRS.. - " ^ 

Measures: 1. Collection of data from participating facilities. 
' '2. Analysis of results^. * 



3. To prepare a report on the results of the fieldxtest. 




Measure:, 1. ^Preparation of a comprehensive? report On the 
V Pa. BVR field test of the FRS. 

Goal 04 : To build and develop ne& evaluation capacities. x 

Objectives : 

K To develop plans for new evaluation capacity in. Pa. feVR.> 

Measures: 1. Completion- of plans for new evaluation projects. 
«*. - * ■ 

* tf * * 

2. Cpmpletion of plans for staff training activities. 

>&> 

2. To conduct projects entailing the development of new -evaluation 
capacities. , ^ 

. . Measures: 1. Number of projects conducted. 

/ 

* 2. New capacities developed through the projects. „ 

. 3'. To have Model Evaluation Unit staff attend selected stiff graining 
programs. ? < * - ; 

Measures: 1. Number of 'staff attending staff training programs. 

2. Subject areas in which staff received training^ 

, 4. To prepare a report describing the new evaluation capacities 

developed'-in Pa. BVR. - 

- v : < 

Measure: H CompTetion of reports > • 

- * } 

£oal Q5i e To develop internal communication, dissemination and utilization 
networks. 

> • - - • 

Objectives : * 

1. To-assess the efferent communication networks within Pa. BVR. 
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Measure: Conduct of study ^of current MIS and agency 
decision-making practices. 

2. To prepare plans for improving the current agency MIS. 

of "pi 
S. \ 



Measure: Completion of^lans for improvement of the 
Pa* BVR MIS, 



3. To prepare 'plans for improving ,^the dissemination and 
utilization of the results of ^valuation activities. 

Measure: Completion of plans for^the systematic dis- 

. t semi-nation of the^tesults of evaluation studies. 

4. To prepare reports describing Pa. BVR plans ^and activities 
related to the internal dissemination of agency evaluation 
activities . * 

Measure: Completion of report. 

Goal 06 : To develop external communication, dissemination and utilization 
networks. w 



Objectives : ^ 



vl. To develop a comprehensive plan for the external dis- 
semination and utilization of the results of the MEU project* 

Measure: Completion of the plan. * ^ 

2. To develop articles for publication in national periodicals 
and "journals. 

Measures: 1. Number of articles sdbmitted/published . 

2. Subject areas addressed in articles. 

3. To coordinate and/or participate in conferences at which 
information regarding the MEU project is discussed. 

Measures: 1. Number of ^conferences attended »at which, 
* . presentations about Pa. BVR MEU activities 

, were made. „ 



,2. -Organizations sponsoring the conferences 
attended. 



Goal »07 : To develop olans for continuing the MEU after .the termination 
of the thre*-^ear contract period. 



Objectives : 

1. To create permanent positions in Pa. BVR for all MEU staf£ 

Measure: Number of MEU staff who have permanent 'job 
slots. : • 
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2. To develop jflans for the continued activity of the 'MEU 
after contract termination* 

Measures: 1. Organizational location of ' the- MEU 
(Program Evaluation* Section) 


c 




2. Numbet* of projects planned. ' 


*' 




3. Subject areas of projects planned. 


■» 
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PROJECT REPORTING FORM: PLANNING, continued 



LINE NO, 



4 . - 

4.10' LIMITATIONS 10 ACHIEVEMENT • , 4.11 

oi: Difficulty in recruiting qualified staff 



MCT1IODS OF ADDRESSING LIMITATION 



Train existing staffs recruit qualified people from 



Hue to state hiring fre eze, state 
hir i ng p o licy . . 



02: Dela ys resulting from administrative procedur es 
'associated with hiring, purchase of required 



' equipment and other start- up actiyifrieg* 

03: Timely awardihfeby RSA of other .contracts 

required to support MEU activities 

04; Difficulty in changing the organizational 
locatio n of the Program Evaluation Action* 



within Pa. RVR. uap ronaul fflnffl < and other eYternaf 

W o ^ O • 

sources of expertise. 

Initi ate necessary paperwork immediately, clpsely mon itor 
administrative process, limit^forch ases and acquisieio ns 
to essentials. ' *. - 



Build flexibility into time schedules which depend upo n 
other federal mntractors, develop rlnae working group 



among MEU states , 



Establish effective informed communication channels * 



f 



os: Difficulty in getting Pa*. BVR managers* and Involve managers and administrators in d esign and 



administrators to change their exiting 
pAt-terna of information u se. _ 



~7 



analysis of evaluation ac tivities, select studies and 
projer f ft o^relevanre . rn their neeHfl . 



06: 



f ATTACH ADDITIONAL PACHS A3 NKI'J)IU>) 



PROJECT RETORTING FORM: PLANNING, continued 



STO. 
NO. 



LINE NO. 



•J3 



fe ERJC 



STAfF INIORHATION 
S.JO JOB TM IX 

Administrator 



s.n 

NIJMBl'K 



Program Analyst 



Management Analyst. 



Statistician 



A^niniff^rfltiY e Aaaistant 



Cler k/Stenographer 
cierK/iypist 



5.12 QUALIFI CATIONS 

Varied e rr* 1 "?*"?? , in ftfiaifiLp 
delivery of vocational rehabili- 

tation; supervisory experience 



S.U RKSPOSSIBtUTlES 

Direct staff of Program Evaluation 



Section 



Knowle dge of analysis and measure- ^ 
ment techniques in evaluation 



Knnulgri gfi nf F.DP fr. j> rnrram , 

Evaluatio n structure .and manage* 

ment'procedures 



Background in statistical work 



JlTftterfttandinff and j 
_administxatiye_And- management 



functions nf PR Unit 



Ability to 'perform satisfactory 
office-related func t ions 



4 



(ATTACH ADDITIONAL PARIS AS NFJiDDD) 



Assist in collection, preparation, 
anal ysis & presentatio n of appropriate 
evaluation work. 

Prepare reports of requested informa t ion ; 
im plement new reporting requiremen ts & 
coordinate activities of MI S-EDP_work. . 
Accumulate & an alyze statistics; tra ck 
budget funds & construct appropriate ^ 

statistical forms 

Maintain and control technical" assis tance 

> operatio n ; conducts researc h & maintains . 
control of flow of information in Unit 

Type regular reports; assist in the control 
^> of records & distributi on; certain jobs 
requ ire communic ajt^n^l^^^injo rmatio nal 
recording. 
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PHOJbCT RCrOKTING t-ORM: PLANNING; continuij " 



PRUjrcT ui.sErs austkact To develop a model program and policy system to fielA test the Revised VR Standards 
^and the Facilities Reportiivg^y3tem and to disseminate the results of this project to other state VR 

Agencies and rehabilitation programs. : : 




. , _ X- 

: — ^ , . . . ■ 2 % 

kc.earch hypotheses amMe^gn m appiicabiei. ( 0 That an evaluation unit can effectively provide information to 
agency staff that will improve the quality of policy and program decisions and, thereby, of the services 



provided; (2) that federal reporting and evaluation systems c an be effectively field t ested in a State 
V R agency. , ' _ 



; * r> 

i\ma collection or recordkeeping plan Pa. BVR MEU Operational Tracking Syste m, studies of the effectivene ss of 

P ES (Program Evaluation Section) p ro j ec ts and the use , of information produced by the PES * 

i * 



An alyses and results; ftonthly progress reports, quarterly reports to Project Advisory Committee, annual and 
final reports to RSA, presentations at con fere nces, articles prepared for publication. ; 



(ATTADI AimiTIONAl. PAGtlS AS NCBI!:!)) 
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PROJECT REPORTING FORM: PLANNING, -continued 




.MO 



PCTAHEP WORK PLAN 



' task oij PlAnpj qR and Organization 

Title 



Oc tober 1979 - September 1981 



7 . II St a it -up and dtira^oii 



7.12 uct>cii|it.ou of ta%k. Establish liaison with Proiect Officer: revise p r oj ect plan and sche dule as needed ; 
attend bi-monthly meetings with Project Officer and staff from other MEU's; establish and maintain 



contact with Advisory Committee ; 'submit monthl y progress reports , 

■ jr* * - * 



7.13 Stjff required: 



7.14 Task result (?) or deiiverjMo(s) : Revis ed work plan and 
schedule ; monthly progress reports 



7.10 



task 02 : Technical Assistance 



October 1980 - September 1981 * 



Title - ^ 7 -H Start-up and duration ^ 

7.12 Description of ta»k. To plan for and utilize technical assistance t o be proyided'by the Coordinating 



Contractor (West Virginia Reg fifirrtl^ml Training Center) 



7.13 Staff required: 



7.14 Task result (s) or dcliverablc(s): Provision o f TA to Pa» BV R 
MP.U BUff ■ . ■ 



7 io 9 Working relationships with Associated 

task D3. Organizations* Contractors & University Resources 

- TT tlc 7 - 11 Start-up and Uuratio 



7.12 Description of ta*L: Establish contact with contractor and other resources working in the area of VR 

4 

pro gram planning and j^lu^oj^_Jjafilu^iifi Be rk el ey Planning As s or Mfl t P fl , W a lker .i^A fl B Ofii a tea , Abx_ 

I xssnclat^ R&T Onr*"-, RAT c^nt-Pr /llni vaxAiX y nf M ic h i g a n . RRI. N ational 

Association of Rehabilitation Facilities, ICD-RUL f 

7.13 staff retired: * _ ' - 7,14 Tj,si rMM,t W or Jciwcriih]<?(<) : Established J.ines_qf 

Jfo^injcatipn and working relationships m 




(ATTAiai APIHTIONAI. PAOHS AS MIXDMO 
/ 
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I'RCUCCr RCPOttTlNG FORM: PUHMNG, continued 



STD. 
NO. 



tlNC NO. 



Title 0 f MEU 



7.12 ucc.pnon of task: Prepare and revise /updat e an Evaluation Plan. (Con ce ptual Framework for Evaluation in 
pa. BVR): p repare and u pdate an O perati onal Tracking Svstem_ 



7.13 Stuff required: 



7.u Tmi UIO or dciiyrabie(s): Revised Conceptual Framework, 

Rfivift'H n ? »™Hrmfll Trying System _ 



tIsk 02- Field* Testing of th e New VR Standards _ 

pz. § n Start-ll „ and Juration • 

Title f * ' 

' i.x 3 oescr.ption of Plan f&,? conduct and analy ze the results of a field test of the Revised VR. 



Standards developed by BPA 



7.13 - Staff rcquircd:_ 



7.14 Task rcsult(s) or deliverable (s) : Analysis of field te s(f*bf 
New VR Standards : - 



7.10 



Field Testing of the. Facilities 



task 03: Repo rting* System 
YYtTo " 



— ■ 7.11 Start-up and duration 

uc*c,. P tl. W of Plan for. 'con duct/coordinate and' analyze the results of a field t est j>f_tha_ 

New Facilities Reporting Syst em — 



T 



7. 1 J Staff rcqui r'cJ:_ 



7.14 Tn^k re?Mlt(») or del . «enhle(») : J^alysis_of JleJLd_t«8£_Of' 



vnniM Hps Reporting System, 



(.VrTAfll AIJIIITIONAI. paoks as Nurmii) 



9 
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PROJECT RCPORTINC FORM: PLANNING, continued 



? DCTAI LCO WQKK PLAN 



tasl »i Dev elopment of New Evaluatio n Capacity — 

SLU- — * ■ 7.11 St ait -up uud duration 

1 1 1 1 o 



, of Desien'new evaluation systems, procedures , method o lo gies; conduct evaluation studie s; 

develop staff ^abilities : dev el op library and other information resources. . 



,.,3 Staff rcu„,rcd:_ >»< - ucUvcrab.cO, :^^valua tion systems an d 

pmcpriiirpa; report on new evaluation capacity 



™* M; - Di s semination and Utilization ^ ^ ^ wd Jur3||on ; 

7.12. oc,cr,p«. 0 .. of ust ! : Improve" the rnrprn^l communication networ ks for dissemination of e val uation infor- _ 
pTinn in Pa. B VR: prepare articles for wt-H^rion in national periodical; participate , i njutional 
rn ^.r fr «. nt-h Pr me P M-n gB at wh ir-h W W nroierr work in djBra i aaea . - 

Staff 1 714 r " U,t(5) " ArtiCU8S <? . » f * r * nC ^ re * 0rt 

on plans for internal and external dis semination 

■ ■ r 

task 03- Program Evaluation — - ■ 

?Ar K -fftte 7ii Start -iip and duration 

7.12 cr.pUo of ua: Prepare, an eval uation of the success of the MEU_ projept 



7.13 Staff required: „ — — — 



MEU project. 



(ATTACH AIwAtIONAL PACKS A<PNi:i WO) ^ p 



I'ROJCCf RCPORtlNG 60RM: PUNNING, continued 



L1NC NO. 



7.10 



UCTAflXD WORK PLAN 



TASK 01: 



Title 



/ Yearly Reports 



7.11 Start-up and duration 



K 7.12 licbci tpt ion of taskT 

4 



Prepare annual reports for RSA which detail the project p rogress and result s 



7.13 Staff required: 



7.10 

TASK 02 : 



Title 



7.12 Description of task: 



7.13 ..Staff required: 



7. 14 ^ Task rcsultU^or deliverable^); Ann ual TepOrt8 



y 7,11 Start-up and duration 



7.14 Task rcsutt(s) or dclivcrable(s); 



7.10 

TASK 03: 

tTtTc" 



7. II S£art~iip and duration 



v. 



7.12 Description of task : 



7. 13 Staff required: 



7,14 T.nk re«i!t(<0 or del ivcrnhlef*) : 



(ATTACH ADDITIONAL PAfllJS AS Nlirmn) 
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PROJECT Rt PORTING IORM: PLANNING, continued 



IINE NO. 
' 7.20 



■ i 



PROGRAM I' LAN JIAHT 



EMC 




See ?a. BVR proposals and OTS for details. It should be noted that the time schedules 
originally submitted were substantially modified during the course ofthe contract. 
Therefore, since this entire^evaluation is being completed at tte end of the three-year 
project period, it was felt that detailed presentation of the ordinal schedules and 
the associated modifications would be of little value now. w 



( 



\ 



(ATTACH AINUTIONAL PAKUS AS NEEDUO) 
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VMUia RCPORTINC FORM: PiANNtNG, continued 



UTILIZATION AWU niSSmimTtrui 

RSA t ' , 



■* * 
Advisory Committee 

• *•♦-• » V 

^ State VR ataff,. general public 



•i»..'0 I'INAI. HA1|jHIALS AMD BMrfAITg .. ,. M .. I >RY ^,6,^ 

RSA * \ y 

J. . 3 

• State yR etaft^ general public 



5 





Telephone 
Meetings 
Progress Reports 
Task Reports 
Annual Reports 

Progress Reports 

Publications /articles 
Conference 



f.12 HMD - SCHCWn: 

As needed 
, Bi-monthly 
Monthly 
As specified n 
End* of year 1 , year 2 

Quarterly 

As sAcified ^ 
■ ^ ThircTyear 



9.21 ' MEDIA 

V. 

a Task Reports 
Final Report 

Articles . 



9,22 I!IS!^^i'>yu 



As specified 
% September 1981 

As they , occur 



Darf . . . "° <-ney , occur 

Part ic^tion ^ relevant As they occur 
conferences and training - 
sessions 
^elected reports 



As they occur 



> ' {AITAUf AINIITIONAI. I7U3.S AS Nl:|!l>l.llj 
* 



9 

ERJC 



82 



v. 



c 



v.? * ( ' ■ 



10. KO. 



u ro 

• rife 



* - " 



M.U 



KB? ' - " 



•i>cov ration or tMi.,»c usjMjct* *f Ocmk» ,flli^._^YgV^^!>tr4 c t w ith §fr&tg V R Ag 6r.cy .te" d evelop .Mai?.dg'eai^^ Information 
£p£&L4ftl ^aiuatir^^^abill ties :_i«J^Joj2mtj!jL of e t af fc-capa bl i i file's to per f ora^waTua t ior, functions 
** : ^fl££J^^ eV aluation 
ijvfltesaa* _ ; • s £ 



itaftiS££&f st»& J J 1 1 ccJiiiTqufes : lope d inc r a assd PA VR F 8S staff c apab i 1 3 ia s\ 



X^-«'!JlUi^ tojiasist in~prbgram 

ipjl^lfegMi _ .4. 0 A ft Sty icq ^a H^JCYj _ » ^ - v / • " . 4 • ^ 

v , * * « v 

'4jL&ySXaiJi»&|RS^n Q //«9tjve%*;s "wf PES? * 

^jia^'prx^ajJi^initL^ ^ * « ll_ 

ifct ^lii f i»proy w „i>o B Vtt*yitc»:^^ agEropriateness of informa- 



2 vA l IA?2^!°£^ 5sed to agaiat 

• ■ . fl * », • ■ -%k\ " : T~. : — 1 

them; to perform their job's. 



V 3i -?- l 5?S?. 1 5-255y^ promoting a more 

\coordihatecT^ppr^^ to problem- fo|ving, and J:he management of aervice delivery'. 



8 



pERJCt 



ro. ko. 

15 



->2 
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PR0.ILCT KCIMWTINC rORM: OUTCOML, continued 



vn pkooram coal slkvlu 
1, Efficiency 



Impact 



15.20 



.1 . Compliance 



. Efficiency 
JL Tmpaet 1 — 



15.11 AITS ft RHGS. 



POS StBLlP IMPACT ON VK COALS 



:8o> 




15.12 Dcscrintion of impacts (positive »rJ negative) 



ncgativej v 
Highes ^quality information and improved use enables bet ter' 
decisioris^r epj arding the > allocatio n *o£ scarce resources. 
* Hi gher oua ljry jnl pj C B &ti oj i m proved us e enfrblqa the 
most: appropriate services to be sele cted f pr ^hg^client and 
jhe gwatejSt.j^sjib^^^ be je alized ^ 



15.21 UcsCTlption of impacts (positive^aiul' ncga^«) 
Hi gher quality, information will tac h^^ej^vaved^^ 
.asje^ment^of progra m compl iance^jLthkre^. Jga^dtti&t 
In insuring _^^nli^cfij — , 

( See abo ve'- line 1 5ilC^) 

. C£ee_ab.avA..-1i tip 1i:10? 1 



: 186 

- f 



( 



* " is" x > 



«a m am) olujcn vi a ttainment Goal 01: To develop a model of a comprehensive program and policy system which 
A^liCf'S&^Sjy uses evaluation information to manage ment. Develop a strong evaluation* un it that is able to 
'assess management .information needs and provide information— co management.* ~ 
v io. n Waiurci jnd Uck»cc of success- See Goal 01, Obje ctiv e 3; Goal 0 4 ». Objectives 1,2,3, 4; f a 

Goal 05, Object ive 2 ^ , , 



A chievement §02 Insure that the Evaluation Unit is in an organizational position which* affords ready access . 
to the Director and other top agency administrators. ■» 

lb. 11 Mv/iAirc* and degree ot* siul-c^s:T ^o ftd ffiini gtrat or Of th 9 PES r apoftg to the Aflttiafant X)irt>nmr rtf f?Anf. rfl1 



« 



*" ^perarl^n g wh o reports t° the Director of PA BVR. Kf/ prti vfr informal rnmrniini ration linfcs and 

* wo rking relationships have been established with all other key PA BVR managers. 

; - — w t " : 

/ * - * ' • ^ - 

Achievement ip3 Employ qualified staff in the MEU ; 9 * ' : 



16 J I Measures and degree of socccssi The PES currently includes 6 professionals and 5 s u pport staff. Pro- 
fesdjonal staff have background in^VR counseling. Computers J Account ii^ ^nd St atistics* ^ x 



A chievement tQA Conduct special stu di es antUeroiectd for PA BVR. in addition to y those requ ired for federal ( * ^ 

^ % reporting purposes.* . V 

16.11 MoDSurc.imd degree of <ucces<; The JESj^as conducted many and v aried .evaluation ^projects during the__ V 

three years ofrthe MEU p rojec t . F or d etails, see t he del iverables in theFinal Report un der A * 

" ' Task 7 - Development of New Evaluation Cap acity* and in t he appendices* ^ 8$ 



(AHAQI 'AimfriONAL VmiS AS.Nf.WlO v-- . 



» * 



T 



..OKI wwwm' ■ ■»«•" wnCOMT., continued 



... IlIU vi. J 



J6 I 16.^0 



V 



ffllL AM) 0M1CTI VI. A ITAIKMENT 

All i, r circfctftu Develop a conceptual framework for program evaluation within the agency 

lo.ir treasure* jnj decree of success: A preliminary version of the con ce ptual framework was submitted in the 
original proposal; gnd subsequent revisions were submitted to RSA, as required in the terms of 
th e contract. < , - 



16.10 



Achievement 102 



CO 



lo.ll Measures and degree or* since**: 



16.10 



Ach i c ve »ciit*05_*_ 



f 



16.11 Measures and degree of sui:ccss:_ 



P 



16.10 



Achievement «4 



'A 

An 




\ 16,1!,* Measure nnd dugroc of success 



4 



| (attach aihutionai; wnus as Nnynui) . ; , 



i0 



I2U VI 



16.10 



16.10 



>16.10 



i* m am oaiK-Ti vi. a ttai rouanr Goal 02; Field test- the revised VR standards and evaluate their effectiveness for 
PA BVR management, 

Atfhicvtwflt 'ulCnmplpfP plana fnr f^1ii t-A«Mn g rh* t, q » pf^Ht fn P a ..-BVR.. — 



io.il Measure* and degree of success: Working relationships were establ i shed with BPA staff; JPES. staff ve re_ 
\ tra V ned iS^H!iS techniques for collecting field te8t*information; follow-up questionnaires were , 
r edesigned; and a report describing PA BVR plans for the field test was* prepared. ' 

: * , - - ' I, ■ 

\ - 

A cMcvcgcnt iQ2 Conduct the field test of the revised VR standards » ^ 

io.il Measures and degree or' <utecss: The closure and follow-up surveys were' completed Mpequired program da ta 
were cdlleTcted; field test results were atialyzeg f^and'data for th* mine* projerr firanrlarrlfl ; 



were collected frcrtn project n&ler.teA f or the pretext. 



A 



* Achievement ip3 Prepart reports on. fine results of the field test. 



16,14 Measures und deuroc of success: Reports describ in g PA BVR's analysis of the field test as well as the 
field test data and findings were prepared. ^ \, 



16. 10 




16.11 Measure uml'rfugroc of sticcctJ t * 



v a " A 



92 : 



(ATTAClf ADDITIONAL |»AlH!S.AS N(!W)l!l)f 



\».u».» mwii«! »•»"" tmtwr, continued , 



- 

4 



Goal 03: Field test the Facilities ~Repor ting System and evaluate its effectiveness for 



a m and oiuicrt VL A TTAiKMirNT *A VR managment. 

Awtno-faa-iit »ui ' Complete plans for field testing the Facilities Reporting System 



lo.n Mci*.rc S and dc K re« of *w<css: Facilities w ere se lected to part icipate: but 0MB approval. vas_jieyer 

ro.ftftivcd for the FRS for mj\ Theref ore. PA BVR decided not to par ticipate, in the^ field test, y 



[\ c\*t?Mn t *Q2 Coordinate tfre field test of the FRS. 

^ ; - - , 



n,.il McMsirrcs and degree of <hci:c^: PA VR did not participate in the field test (see above). 

5 ■ — % ^^Fe 



; ; % 



Achievement 103- Prepare a report on the results of the field testt 



16. 1 L Measures and "degree of success: PA VR did not participate in the fi eld test (see above) ._. 

• * 
( Also, see appendix 13^in Final Report) , — t : — _ 



7t 



Av'lilcVCMIlt *M* 



10.11 Measure and degree uf^ success: ^ 



— t — 7. tr 



*~ 



HAmdirATiiHf lONAt pai& as nBcokio, 



94 



- r * 



#».UI* t A'JJMJff fiMM;. Of/TUMli:, continued 



* * " * Goal' 04: Build and develop new evaluation^caoacities 
ca m aw oiuicti vk a ttaikmknt «va*u» l - 4W « capacities , 

AciactygcMt hii Develo p plans for new evaluation capacity in PAVR. 



«.«ur„. aiW dv K rcc of »ucccsk:J^s^ activities atd gWf ^n ft were 
- J — ■ ~= prepared, < . 



Acl.cve.cnt «02Conduct projects enr,rt1,> rnn dnvn1npm,nt of n.u ^in.Hnn eaM „>w ' / » 

M..JI M««inr.^ Jv fi rcu of ^cs^Thirtee^ 13) projects and activities w.r P r 0 mP i e ted by the ggg -(see 

^ " A n al ys i s T ech n i que , F nl low^ Uj-^xudika, . Manaoemenr Tifon-f^ 'cyo^ qj-n- Benefit*, rii mr 

SafisfarHnn SMiriips , ^ f,„i„^,-»„ rtf ' rhr ri i rn t ftt i 'i B tai i LU 



Achievement «0 3 HavefMEtT s taf f attend selected training, programs . 



I6.ll Insures und Uc K reo of success es staff attended training programs ahd/or receiv ed training in Program- 
Evaluation and Review Techniques (PER T ), CriE ical Path Me^s (C>A). Pro po^T writing- SAS jCo *- 
puterj^njuage), ttB^tjM Analysi s, Cost-Benefit Analysis. 'EasvtrW ^Compujer, language) 



„ " . / L 



Airhl went KM Prepare- a report de sc ribing new evaluation capacities developed in PAVR 



lu.li Moiisure nnd Uugrcv ttf success: A report was devplnnprf ar*A fc u ^ ^ . . , 

.^gHorc wag_ge ve ioped »,_^LlLLglU^XjO^J±€L^^, capac ifciea.axe . induced 

JjL^ejjdic^8_14_andl5 - Fin a) ^porf, t . . Y ■ ^ 

' * " " a • * — " 1 =r — * 



lo.ui. i « whims 



Otmw*:. continued 



Goal 06: Develop external dissemination and utilization, network. 
Develop a comprehensive plan for "the' external* dissemination and u tilization of the results of 



AclllCtffK'Ht >ul 



the MEU project. 

w,.n MM»rt» unJ Jc K rcc of mccs>: A plan for external' dissemination was prepared in conjuncti^n^th _ 

• \ , 

ffifcr MEU 1 f w<\ Vat Vir ginia R esearch & Training .Ceotex«__ „ I, ; 



aJUf 



Achiwcnt 102 Develop articles for publication in nationa l periodicals and journals. 



r 



lu.li insures and dc fi rcc of ^ ^ .Tfte following artic l es have been prepared: ^ Ca8e_Rev xew Process in 

Em gxan Evaluations I'Wufltion sect ion Library"' " T h e ModpT F . vfllufltion Pro j ect in Pennsylv ania", 

nn.^n^/o'f * MoHpI Pv fl Sn fl M'nn . TMir" and, "Working KpI atianihips flfl a Farrnr in rhp 
♦ > • • it 

Durability of Program Evaluation « ( • : : . : 



i*n rnnforenc+^A t wHinh i nf nrmflHmrf regarding fhp MRU PrO J Pr ., t 1 *. 



16.11 



'discussed « v — '* t / 
! Hv« U r« -nd dc U rcc of PA BVR ataf f have p a rticipated in and presented in the following • 

' conferences! Region III Program Evaluation 'ForumsC Mana^ m^t uses of Evaluation information. - 

»PA in Spoft-*da1p t ' A7 f RiRter- Srfl<-A« Cnnfv>rPncft hosted DV-EA-MRj r& 



16.1 



sure nnd dogrc^ of lueces*;^ 



9 a 



(mMUAimiTlONAt PAUliS AS NOTf!!)) # 



• \ 



f ».Uf«f MlvwllwI'Wf: OmcW, continued 



_ 



5 XJoal'O?: Develop plans for continuing the MEU after t^e termination of the three-year 

i^M^t&H^ Vfi. A rrAiKtOT- contract period* ■ i 

Create permanent positions in ?A BVR for all MEU staff. 



JO 



lo.n ikMsure* and du K iyc of succc»^: jAll MfcU staff have permanent pos it ions in PA BVR, 



A Jn&vtttfiu >02 Develop plans for the continued activity of the MEU after contract termination. 



10 



10 



n».l! rh.us ires and degree of «uuc<>' The PES will continue to operate after contract termination; "particul ar 
~" ^O^^ibilities mil include preparation of the evaluation romponpnt of agpnry prnjgrra, 

and TnVPHtigat-.tni'l nf flPlprrpH fl.gppnfq nf/ program npp r al-jr> n g . 



Achievement *03 



16.U Hoa»urcs undt degree of success; 



AcIucvcbciK i04 




lb. 1 1 , Hcunic ,tnd doKfce o£ success 



1 '*\ 







100 



(ATTACH AIMITIONAI, P.MJliS AS HUVtiVM) 



00 

o 



PR04CCT REPORTING Kftttf: QUTC&t. continued 



.D. tiO. 



17 . 



•' -in'1' 



LINE NO. 



17.10 



ProbUn dmcrtption: .Receipt of required support from Federal contractors. Contract I to private organisations 
for conducting t he VR Standards field test, the FRS field test and providing coordination, supported TX 

.tn ffte MEUlfl tfPTO fltfardprf up fn nnp ygn r lgtET than anHr^pafprf . • " • , - y « \ 

* ^ : ; — : — T 1 ' ! ; — : " — ; r*r- 

' ' » ^ * * h ' " ^ W" 

RccutttenJed remedy: Design flexible schedules^ so other activities could be performed ^during periodT of waiting 

fpr the Federal 'con tracts to be awarded; d q, not; avar^ one ' contract and deaign'tfl^s whioh ge j ^litfl th e : 

y ■ ^ ■ . . f ^ - i * - r , v • ■ 

support, o f o fh er contr ac t s wh ic h -ha ve nnt yet been aw a rde d ^ - ^ . ' * 9 ^ 



17.10 



17.11 



PROBLm IDlJmnfcATfOH 



*\. • • . . • • • 

• • V ' . - ... ■ f . : r , 

Probicn description: D eveloping lystem for \ insuring that evaluation*, information is .used by agency m a na)tera, and 



RccoMcmtod rcWJy: I nvolv e staff i n selection qf topics for studies, designs and analysis ;- r eport resul ts' in a 



glmel^ manner; make reports concise and easy to understand; Include specific instructions for how to use 



' th e results presented i n the report. 



* , PH0JECT WvPORTlNU FOHM: OCTTCOfE," continued 



H 



if > „ • 



LINE NO. 



ir.io 



••17.11 



17.10 



17.11 



PROB LEM iDKHTlFlCiVf fON - ■ ; - * fc „ 

v : * , s $ 1 

ProbicB jticription: Project delays resulting from administrative procedures associated with hiring, purchasing 



of ^required equipment and> o thefr start-up activities. 



KvcujMcnJcJ rc^cJy: Allow adequate time^for such delays in initial work schedules: start all required admini stra- 
te . ' , 
n v » nroceaaea » t simn-laa: paaa^hlfej *n thorough r»*»»r<*h tn ^fArminp Hip aHmihittt-rativp requirements which 

must be, satisfied for a given project. * ; : ; , 



• Prjblca description: 



RccoaiicnuVd rcacdy: 



10 J 



V 



6 > V 



*.«'* 



{4 
iiuih.lt 



hbrokTiwi iokm: cxrrcoMi:, continued 



Uy cUwut of Jc»ij;n aud opirr.ittoti.il procedure*, nunpower need*, 'tine!, mid com i^ipii results (incliidiiii> discussion of Portlier ret mmx 
ninco l.i*r progress ^report): * . 

\ * , 

Same as initial design. See lines 5.10 and 5.20. Manpower needs will vary depending upon the support s 

which an agency, desires from its evaluation unit. Usually, however, budget constraints dictate the manpower 

which £ t an be used r and the Unit's activities are planned based upon the manpower, available . 



* . .|ATTAM AlHlltllWAI JUKI'S 'AS WilMitt) 



.PWV1ECT KCCOrTIW; FORM: OUTOWU, continued 



>f(V NO-. 
' 18 



00* 
CO 



LINE NO, 
18.11 



. REVIS IONS SINCU UV ST PROCRrSS RIIPORT AN1I OR IGIN AL DESIGN 

•None. . . ' 



r 7 * 



.v " **** 



r * ** 



m ■ 



waiter airoRnxu 



j. so. 



20 , 



SET*.* 



LINE NO. 



20.JU 



iniUZffWWMNll I) SSKMNATION 



w 



iNlllVIWAfivCROfo'S i:OHTACTi:D AHff* 

tfunrlixis oi iicuiirr iiuuncs 



PSA 



Advisory Coitmittee^ 



Staff from other 



State VR ag< ncies / \ 



I'ORM; OUTCWi:; continued 



20. i-j m)im 

Sane as 'originally 
- reported 



Same ag originally 



20. J 2 H^imArK 



Not yet received 



reported 



Same as originally ' * 
reported » 



N o formal, fee dbac k 'received; apparently work 
was satisfactory v 



Inter/st was expressed in th e m new syst em s and 
techniques discussed. 




01 




• J—7T- ' ~ 
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» 












J 

1 




X 


1 » 
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♦ VI.' EFFECTIVENESS AND EFFICIENCY OFVTHE MODEL EVALUATION UNIT 

.-AND ITS .INJJCRFACE'WITH OTHER .'PROGRAM UJIITS . ' ' * . 

The.- Program Evaluation Sectiqn (PES) interfaces in multiple" ways 

* ' . - . 

with the Central Office sections and the district field offices. This 
spirit of cooperation has always been present and has* been enhanced with 
;the development of «th^ Model Evaluation .Ufcit. • 

Ipteractions and activities/increased with the development, of threl 



new activities : • • • 

1 .* Technical Assistance, Center (TAC) The Program Evaluation 

» * ' 

Technical Assistance Center was established as a Result of . 
Pennsylvania BVR receiving this federal contract, which mandated 
that 'a .library dealing with program evaluation ,be developed. • , 
However, as the work progressed it became apparent that' there 

• 7/ 
- . / * . 

w^s a need f f or a mare extensive library whach would serve our 
entire Bureau. 1 As a result, the Program Evaluation Technical 

. / Z ' ' 

Assistance Center has approximately (2,100 holdings covering many 
areas <of vocational rehabilitatobn including new* rehabilitation^^ 

■ . " .■ V > . * , 

^ * trends, current legislation^ specific disabillrtie^^assistive 



devices, etc. Since its inception in Janu^r^-WTV , theTIibrary 



has disseminated over 1,600 articles and books 'to the staff of =- 
the Pennsylvania BVR for their use. It has proVen to be a 
useful source of ( information for our staff and a means by which, 
they can obtain, current information in ^rhe fifld of rehabilitation 



Appendix 16 gives details on the methodology used to establish 
^ # the Program Evaluation Technical Assistance Center, the services^ 

; n * \t provides, and the means by which material is -disseminated - 

- to the users. > 

* • . * * 

2. Program Evaluation Section Goals and Objectives Manual - The purpo 



.of developing this manual 'was to give top management, or any 
exogenous source the knowledge -of what* our program Evaluation 
Section,, evaluates and how we measure quantitatively our \ v 
objectives. Thi$ manual, ('Appendix 8) is *a f 4" escr iptipn of 

current activities conducted in the Program Evaluation Section* 

* • * . * ' 

^ The go,als and objectives stated in -this manual reflect work 

which involves liaison with Central Office/field office staff, 

* • .1 

federal/state agencies, and other external sources. ^ . 

3. Program Evaluation Section Brochure - The Program' Evaluation 
Section brochure (Appendix 27) was;\developed for use by the 
Bureau staff describing how Central Office and district field 

iices can use., the Program* Evaluation Section' as a supportive- 
agents. .Tite^ES -brochure describes- the -following : t 
1. What -is Program Evaluation? 
«2. , Who doea it? 

3. Why do i*£ ' fifa . • ' ■ 

. 4. How is it done? i ' * " 

5. Examples of what Program Evaluation does. • • 

» ♦ 

6* Examples of questions or problems Program-Evaluation' can 
help v with. . • I 




r — ' 



The following, interactions and activities have occurred or are^ 
continuing with the following Central Office Sections : ^. 

1". EPR Sect^n '- jjain'tain a direct linkage- fpv information requests 
that require,?' computer programming. Tfcie Program Evaluation 
• Section and the EDP Section act as supportive sections to the 
'Central Office and* field offices. 
2. Grants arid Facilities Management Section - Developed program . 
evaluation pl^n for the Client Assistance Project, assisted'in 



- 1 



"86 



111 



- ip the questionnaire design for the collection of dat&^for the 
. fcomprehensive Rehabilitation Center Projjscft, developed a program 

evaluation plan for (tenters for the Independent Living Project, 
^and provided the rehabilitation specialist? with various types 
A of 'information • * ■ 

t . * 

' Public In|oriflfftioiY Section - Write informational articles, provide 
statisticaBl^apta on 'our overall program, .and provide this Section 
with/ihf^iicttion from our Program Evaluation Technical Assistance 
Center 



• vTrain^^yectiop - Worked with this section in providing training ' 
prog^jlpVon Similar Benefits Utilization and the Case Review 

\s3Tf 

Process. We used the Training Section to produce transparencies 

' * . \ 

for our oral .presentations at various meetings, 

Mecfecal- Section -^Supply statistical data on request, e.g. clients 
with spinal cord injuries* * • * 

Legal Section - Provided information on the* consultants usedSV'** 

• • \ ' ■: • c • * 

the Program Evaluation Section , Also utilized this Section for 



questions concerning ^"Confidentiality of Information" release 

by our Agency to external audiences. • 

Affirmative ftction Section - ^Supply statistical data on request 

concerning defiiSgraphic characteristics such as sea/ race, v age, 

i 

and Spanish surnam by counties. 

Special Projects Section Provide statistical data on the Veterans 
■ » * 

Program, 1 No -Fault Insurance Program, Aging Program, and the Drug 
and Alcohol Program. Completed a Program Evaluation Assessment' 
of the Aging, Drug and Alcohol Programs annually ♦ 
Budget Section >- Assist^in the development of the RSA-1 Report 
Program Financial "^.an and State Budget. Work with'the Purchasing 



OfMcer in the ordering of suppliers^ library books .and 

)».; * " * - * ' * * \. 

periodicals for the Program Evaluation'^echnical Assistance ^ " 

Center. . ■ 

• • • . * ; . V 

Systems Development Section - Provide infowhatiori- for the 

Workers 1 Compensation Program^ Placement 'Program, £killsT • 
t n ** " ' * * 

Tuning and Enhanced Placement Program, Deaf ^pd-Har4-of- 

Htear^ing Program, 'CE^tf and' Public Welfare oPrograips, Mental' 

Health and Mental Retardation Program, Johnstown Center* 

Training Program for Employers, and we also aid .the Psychological 



Consultant. 

Fiscal Section Regular interaction for-budgetai?y control 

and work; * * 

Poligy and Procedures Section Development of new forms (BVR-104) 

and development of dissemination- techniques'. 

Case Service Section N - Assist in the development .of the 

definition of severely disabled and informational requests 

from the district offices. Provide information ai^d analysis 

of' similar benefits use by district field of f icfesL ^Provi^e N \ r 
' . / '1. ; X 

detailed analysis and recommendations fallowing all case reviews 



.inqu/nt 



Monators the number; of delinquent annual caseload reviews, which ' 
• ■ • j . • * • . \ 

is part of. the ItyRP ^process . \ t 

Assistant Director of Central Office Operations ~ Provided linking 

references for development of t^e Consumer Advisory Committee. 

* • . t \ m * 

Utilized the Follow-up Surveys for input to this Coira^ittee. In 

addition to these interactions at the Central Office J*evel, 

many contacts are made at" district fiqld office level ,f with *tfie 

Program Evaluation -Section responding directly to "speckf ic 

. - , v • . ' 

requests for information and data. " *.«\ 



Discussion - Which of ' the above were mpst successful aftd why? 
The Technical Assistance Center had a decided impact on our field 
^counselors and supervisors, -,The TAC 'used Afield input in determining 
the types of acquisitions needed for 1:he Center. .The acquisitions 
purchased* were books and periodicals that provided the necessary 
'information to the appropriate users • Another activity that was*- 
extremely, advantageous' in the effect ivenssf and .efficiency of our MEU , 
ctnd theW interface with other sections within Central Office was the 
control for all' informational requests submitted by Central, regional ^ 
and district office personnel, and external soiurces.- 



• VII*. mod£l evaluation unit capability of providing information 

RELEVANT T<5 INTERNAL 'STATE AGENCY PROGRAM 
: * PLANNING AND MANAGEMENT DECISION MAKING. 

' V . ' * , ' \ 

> The main objective was^ to determine- if information currently being 
provided was Meeting the nteeds of decision makers. Considerable effort 
was. spent Reviewing the* Agency's Management Information System. Initially, 
.the Unit prdduced all routine Federal, State, and Agency reports. We also 
conducted a number- of special programmatic studies, - \ 

fhe intention of pur review was to assess the information needs of 
top managers within 'our 'Central Office. We concentrated on those reports 
produced within our Unit. However, those reports produced by our EDP 
* Sect ion were also touched upon. This review was cond acred during the 

* 4 A 

, t 

firsthand sjecond years of our contract. • y 

"° A preliminary format of ,ai) analysis of informat/on requirements 

was developed and is presented in Appendix- 34. Fr^m^this, a data Require-. 

*\ i 
ment analysis was -conducted. The results of the analysis have been 

integrated into a system design which shows 4 the floW of information within 

the entire Agency.- It also' identifies specific information needs, as 

well as information redundancy which existed. A significant number of 

overlaps indata generation or availability were identified. 

. - Each report generated by the Program Evaluation Section was reviewed. 



Appfen^ix- 3S v sliows a flow diagram of the. process used tq analyze 'each 
report. A format far this analysis is' included* in ^pendix 36. The • 
'dissemination and utilization of the reports Was also reviewed using the 
process outlined- in Appendix 37. * To obtain the necessary detail on the 
specific needs arid l uses of the tljS, the format described in Appendix 38 
was employed/' "This process has "been quite productive and has yielded t^e 
data necessary 1 ' to make substahtial revisions to our current system .^Among 
the revisions made were: / 
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Discission - Which of the above were most successful and why? 

V -~ 

The Technical Assistance -Center had a „ decided impact on our field 
counselors and supervisors. The TAC used 'field input in determining^ 
t>ie typfes of acquisitions needed for the Center.' The v acquisitions 
purchased, were books and periodicals that provided the necessary 
information to *the appropriate users. Another activity that was 
extremely advarita^eoms in the effectiyenss and efficiency of oyr MEU 
and their interface 'with, other sections within Central Office was the 
.control for all informational requests submitted by Central, regional > 
and district office personnel, and externa^ sources.- 



vii.' modIl.. evaluation uni! capability of.^peoviding information L 

*. *• RELEVANT TO' INTERNAL STATE* AGENGY PROGRAM" » ' , '. *X 
* PLANNING AND HANAGEMENT DECISION. MAKING . - • , 

*' ': ■ ° . . t ■ . . ' 

The main objective was to determine if information currently b£ing 



> 



T ' . . a v 

provided was meeting the needs of decision^ makers. Considerable effort ^ 

' - a c 0 # 

vfes spent reviewing the Agency's Management Information System/ Initially, 
the Unit proceed all routine Federal, State, and Agency reports, /fte also 

f 3 ' 

conducted a number tf>f spfecial programmatic studies. , . 

* ' " • v , I ' *"* A ' 

l The intention of 6ur review #as to assess the information needs of I 

vt *H ^ - ' . ' ' * * ' ' ; ' ~ 

1 top'manasers within our Central Office f We CQncentraJe'd on those reports - 

'produced withih our 'Unit'. However, those reports ptfodbcetf ,i>y our EDP/-V,., \ . 

* Section were also toilched upon. *This review was conducted during the * 



-V \ 



"first and second years of our contract, 



A preliminary* format o| an analysis of information requirements * * ; ; 
S l was developed, and. is presented ifc 'Appendix 3H.^^^rt^iI, 4 a' data require- # ^ 



the entire Agency. 



raent analysis was'ccftiducted. " The* results \of the 4 analysis 0 have, WeiiJ • . 
integrated into a system design* which shows the f^ow^o^ information within 



It also identifies specific jii£f privation ♦fi^feds > as 




well as information; redundancy which* existed* £ significant number of • 

^overlaps in' data generation or availability were 8 identified. , r * 

/»'.•- "* " r - • 

. Each "rejport gfef^rated,' by the Program * Evaluation Sectiorf was .reviewed. 

Appendix 35' shows/ £ flow diagram of -the process used to analyze each^ b " 4 

report. A -format 'for this\analysis is included iji *A^pendix ^36. « The . 

dissemination and utilization, o.f the reports was also/^^viewed Using the 

rocess outlined in Appendix 37 A Tcr obtain the rieceisary Retail 6n the 

specific need$^ndj us^s W'tWe -Mis-, the format described in Appendix 38 

"was employed. 7 This process hak-^een quite productive -ang Jias yielded the 

data necessary tb.'make substantial revisions to our current jsysteiti. - Among 



1. Reduction in the number of reports produced (from seven to rive)..'*' 
' - ^ » . <* - 

* • ' ■ - ' 5 ' • ' / 

2. w Reduction .of the frequency .(monthly -to quarterly). ^ - 

3. 'Modif icchiion 1 of four reports. J 

4. . Reduction p'£ one report (from t£n pag£s to five page^. 

• * » » 

& . Even tholigh we made these revis'ions,. we fburid that the informational 

: | - \ . 

needs of our Central Office were still being rfiet. - 

During our review, it was deemed necessary to compile* a list' of all 
available ^data reports. Appendix 39 is "a, cojpy. qf this comprehensives list \ 
showing (1) computer run numbers, (2) title bf run, (3) frequency of run, 
i(U) product dissemination, and (5) "distribution 'frequency-. 

At this time, we are proceeding with phase ; two of our analysis 
of the MIS. This phase is similar in method to phase one. However, a 
> wid er b ase is involved as we move toward the level of field operations. 



During this phase, our Regional ^ndmistrict Office Administrators will be 
interviewed. Thg procedure for~these interviews and data analysis are 
'presented in Appendix 40. It is anticipated that, the results , -when 
f„ analyzed, will result in a more streamlined and efficient MIST7 

. ' While phase one and phase two involve only data dissemination ftfojp 0 
the Program Evaluation §ettioQ ^nd/or EDP, in order %o make a comprehensive 
review of all data, services, phases, three will involve data that is 
disseminated by othexi^S'ections erf the Bureau. Although formal planning, 
has .jiot; been developed, it is anticipated that a plan will be developed 
shortly. The plan will include a collection of all data reports that are 
, prepared and disseminated by other sections^ of the Bureau. This collection.., 
^jwiUT provide a 'description of th\ sources of information and utilization 



. of data presented. W^feel that tftis'list yill be of "great help to manage- 
' ment in contacting* the appropriate sources of data. This will also be . j m 
helpful- to management in making work- assignments. . " / 



As part of the first phase of analyzing the MIS, the Program* 
^Evaluation Section formalized a procedure for the request of information. 
During the second ph^se, this process was refined with revisions bein^ 
made to appropriate documents. A flow diagram of the request process is 
presented in Appendix 41, and a narratiye description in Appendix 42. j 
At. the same time,, -a log was set up to maintain .a control, record, ajid 
description of 'all requests (See Appendix 43), 

«% To facilitate better managerial control of the flow of information 
from the Program Evaluation Section and/or EDP Section, a .new review of , 
re que^t^ procedures has been developed and implemented. The procedures 
to be^followed in processing: a request for information are outlined in 

Appendix 44. A flow diagram of the process 'is Appendix 45. The guide- 1 - 

*<? = 

lines used to determine feasibility of a reque c st are presented in 4 
Appendix 46, A revised log book used to maintain control of ^requests " 
has'Tteen developed and is presented in Appendix 47, 

In additidn ta the procedures developed for the request of information 
from,pur Cenfral Office and from other agencies,** a procedure ha^ been^ 
developed jfor our district offices. These -procedures 3re outlined in 
Appendix ty8. ' " . ♦> * ' y 

The following cQmments were received from various administrate!?^/ 

■s — * * " 

decisioq makers 1 in the Bureau oencerning the support they received from 
the Program Evaluation Section, Please refer to the organizational £hart^ 

(Appendix 6) for each- individual ' s location within the organization. 

* \ 

1, Ken Fleming, Coordinator f Job Placement Program, Systeips Develop- > 
ment "Section : "Let me justr outline , three of the'most recent 
areas where we have called upsn the service of our Bureau r s 
- Program Evaluation staff, ' A _ 

^— ; — '-^ "Our agency w^s^granted a two year I £ E grant , named the 

i . 92 ^ 



Skills graining and Enhanced Placement Grant (S,T,E.PV), to • / 
introduce within our Agency some additional" innovativ.^ placement 
techniques. We' called upon the Program Evaluation 4taff, in 
preparing the S.T.E.P. grant, principally in^three areas. We 
asked their assistance, and received their assistance in regard 

% 

to data collection,- data comparison, charts and graphs, ... 

"And we called upon the Program Evaluation Section to assist 
us in designing a format for the reporting of pl&cement counselor 
placements, We have done this rppn-rting on a mrmth]y h^gjg using 
the designs suggested to us by our Program Evaluation staff, 

"One of the' more recent ^uges of program evaluation^ in our ; 
placement program was^in the area of a-case review -that wev ^ : 

' s • - , % - 

Requested, We met with the Program Evaluation staff and asked 
their assistance in determining the appropriateness of vocational 
goals in a random sample of case files,./, L 
"...We then Utilized the results of the evaluate 
designing some placement program modifications to -assist us In 
letter utilizing our EDP capacities to match available clients 
*to existing jdb^rders/' 

David Williams, Administratpr , Planning Branch: !u . . 



ion- in 



I believe- 



that the Section has a major role in the future of the Bureau, 

* c 4 ^ 

4 * / ♦ 

Thi.s is a Section that we depend upon to advise us of several 
important aspects of^lrhe program as they occur, 'The Evaluation 
Section, in providing us with this information, also enables' us 

to project what we would like to see occur in tjie future. 

- <: * ■ — * ' 

Personally, I find myself becoming more and more dependent upon 
the Evaluation' Section for detailed information regarding what 
the Bureau is doing. 



r • 



"•••The Evaluation Sjection measures the overall rehabili- \ 
tation program; "does thi^^y collecting, analyzing, developing 
product information; disseminates information' of -the Agency's ' 
program to the appropriate 'people, 

"•••The Sectibn monitors the vxie^tional rehabilitation 

process, quantitatively^ and Qualitatively 5 f? r achievement of 

goals and priorities* It "c'GrKlycts^ ca^e review studies on, 

specific target operations in the rehabilitation \.B£oces'S,. > It 

;^licits input from &11 organizational levels to develop 

prae^dures for -internal utilization. Through heeds assessment 
.1 ** » 

and utilization of existing reports, it develops a Management 

Information System, The Section provides meaningful information 

-> # . • «.'."" 

to management to assxst other Central Office Sections develop 

information and assess informational n£eds which tfiey may have, 

t 

It continuously builds $nd refines methods and techniques fc>r 
program evaluation. Finally, ^ the Section develops evaluation 
plans for special projects being conducted by other -sections," 
James Diffendal, Rehabilitation Specialist, Facilities and 
Grants Management Section: "Although the Facilities and Grants 1 
Management Section of the Bureau of Vocational Rehabilitation 
has used the Program Evaluation Section onjnany projects in 
the past* the Program Evaluation Section was utilized extensively 
in the collection of data oti operating CAP projects, which was 
then utilized to submit a CAP grant to RSA L by the Bureau and the 

Office of the Visually Handicapped, The g'rant's Submission to ' ^ 

* ' \ 

RSA, however, 1 was not approved when it was f£rst Resented 

' ' r . . \ 

because the Program Evaluation component of £he grant Was 
proposed to be carried out by an outside source, RSA suggested 

■ •. . ' - • .-94 - 

■V • • * 121. ' '. 



utfilTze <^ur i 



that, in order to s^re dollars J we utilize 6ur ihternal Program 
Eval)ia|ion Section. The program Evaluation Section did design 
« the^ instruments and methods of data Collection J and the grant 

- 7 7 / • J • i 

: was approved for funding. WiAhout^the ^development of th^se 
instruments, it is safe to s£y that the bureau/ would no tj have • 
competed successfully for cj? fu^ng. Prior /to the service 
component of the CAP grant ,/ the 'Program Evaluation Section -then 
held a conference ^betweeiT^e BVR administrative straff land our 
Philadelphia and Rosemont offices, andvthe CAP staff themselves, 
V to review and explain all the responsibilities of the parties 

• in the collection of the data for this grarrtScj be successful., 

\ 

The administrators of the' ^urgau felt verV good a^oUt^this explan- 
ation as they felt now that they could understand the CAP grant 
in full. The Program Evaluafc^n individuals have been very 
helpful overall x and we ^hink that they will be important in the 

success of this grant." V 

. > / 

. karlin.Kester, Rehabilitation Specialist, System^' Development 



Section: • * 4^ ^ *" < 

^ "Historically, Pennsylvania \bvR did^not kve a formalized 

relationship with the Pennsylvania Bureau of W&rker's Compensation 

for referral of work- injured persons. However ,\ due to ^ the volume - 

\ ^ 

of morvtjily work injuries knd the continually risW costs of the 
Worker r s' Compensation Program i? Pennsylvania, BVR's interest in 
developing. a more form^J^relationship grew. After initial - 
meetings between the Bureau |directors, the possibility of a. 



V 



.computerized referral exchange pilot '.project was' p 



Program Evaluation Section assisted in the -prelimiiiary* design 
and presentation of the pilot project to -not only 



oposed. The 



:heTrk 



spective* 



Bureau directors, but personnel from the fifteen BVR:district ■ / 

offices who would eventually receive referrals f*om the project. 

This innovative approach to the referral of Worker's Compensation 

recipients has been in place ^ow for V\ months. Thi^Program 

Evaluation Section has Jbeen monitoring the project through 

quarterly statistical reviewl and reports. We have found it 

necessary to make changes in our initial program end reporting 

system during this, time' based upon these reviews.'. Curreptly, the 

Program Evaluation Section is assisting me in a review of the 

totajf^project's impact in the BVR program. The,; Program Evaluation 

Section provides me with ^important staff support. functions .as the 

coordinator of Pennsylvania's BVR Worker's Compensation Referral 
■ 

Project. 11 

Lucy Shjimaker, Rehabilitation" Specialist , Policy and Procedures 

a** 

Sectlbn: " ^ 9 • 

"Often a g&p exists between .the generalities of a mandate 
and. the specifics required fotf field application. This divergence 

became particularly apparent when the Bureau addressed similar 

• \ . . • 

benefits implementation. To bridge this gap, we in Policy and 

\ 

Procedures -called upon the expertise*^ the Program Evaluation 
Sectitfn. W^at we were looking for. was to identify a procedural 
stapting/poiht. ^Through the Program Evaluation unit a case 
rev^e^ v/as Conducted and, from these 'findings, interim guidelines 

were developed. These guidelines provided our fieLd personnel, 

i 

with! a uniform definition of similar bepefits, the specific types 
of resources to^Jbe considered, as well as a method of recording 
resource crediting.. The application of these interim guidelines 
was then' evaluated through another ^ase Review. It was through 

' ' , y | ■ 

' '• "-' 96' 



this cooperative effort that the Bureau's general policy regarding 
similar benefits utilization colild be refined into meaningful 
procedures. The results of our joint endeavor then became the ■ . 
final procedures now established *in. the Bureau's Case Services 
Manual. I would, like* to point out, too, that through this' Program 

Evaluation System, we are able to track the ongoing application 

♦ * *' 

of the final procedures. In 'this manner, we can more readily 
accomodate further revisions as need may arise." 
^AdolphT Latz, Assistant Director, Field Operations: "The Program 
Evaluation .Section at our Central pf fice has been requested to 
supply both management and operational information' We've asked 

them to provide us with dafa related to similar benefits in two 

t , * - 

ways: first, to determine the* costs of similar benefits, and 

<-■' , * % 

secondly, the number of clients that were involved in similar 

benefits programs as a client of the "Agency. District Office's 

have made "several requests for operational ^formation; they've 

o ' * > 

made requests concerning the riutaber and costs in training cases, 
particularly those at technical and trade schools. Also, we've 
asked the Program Evaluation Section to develop impact information 
"on .ceilings for certain expenditures. They have provided for us - 
this information in summary form, which was helpful in making 
decisions, as well as implementing operational procedure^." 



VIEL, NATIONAL BIS SEMI NATION AND UTILIZATION 
* ' ACTIVITIES "DURING THE THIRD YEAR 

The joint external D&U plan was completed in April 1980. Information 

sources, kinds of information, and audience identif icatioa and analysis, 

| *■ ~ « 

as well as vehicles of dissemination were included. The major methods used 

were presentations at meetings, the ''Manuscript of Special Observed 1 
* Activity", >"MEU Newsletter", national -conference, and Sister States Confefr- 
ence. ^ 

Pennsylvania has submitted articles to various publications coordin- 
ated by the coordinating contractor ("Manuscript of Special Observed 
Activity", "MEU Newsletter"; and "MSOA"), Appendix 30, Participation in 
the Federal Region III Program Evaluation Forum held twice in Rehoboth 

[' \ . \ ' 

Beach allows MEU and nota-MEU states to discuss issues of interest in program 

evaluation'. ' ■ . . * 

: • « 
A Pennsylvania MEU staff member, gave* & presentation at the St, Louis 

National Convention of the -American Personnel and Gyidance Association, 

(See Appendix 49). - - ; - 

i 

^Several presentations were given^at the National Conference held in' *- 
Scottsdale, Arizona, on June 30-July 2, 1981 by Pennsylvania personnel. 

The topics wepe .centered on how management utilized .program evaluation 

* ' ' > ' ' ' ^ 

(See Appendix 31), -/ 

Planning for the Sister 



States Conference began in September 1980, 



and progressed exceptionally .well culminating in the September 15-16 

% ft 

* * * * ■ , 

conference. The Sister States Conference was held in Harrisburg and the 

* . V It 

following, states participated: New York, Florida, Ohio, Illinois, Texas/ 
California, arid North Carolina (see Appendix' 32)1 

Pennsylvania is very interested 'in external D6& activities for the 
third year and beyond* It is our hope thart RSA will allow funding for DSD 

4 * 

activities on a national basis at the conclusion df the contract. , 



9SU , 



1251, 



iX. PRESENTATIONAL DIMENSIONS OF | STATE VOCATIONAL REHABILITATION 
9 AGENCY JE^QGRAM AflD POLICY SYSTEM DEVELOPEB WITH THE USE OF A 
MODEL EVALUATION 0NIT/ MANAGEMENT ^FORMAOJION SUPPORT UNIT 

If a Model Evaluation Unit is to hchte impact on the Agency's 

policy making, planning and program Operations and be -accepted within 

the Agency as being important to the! Agency's concerns and needs, 

meaningful information that has utilitarian value must be provided by 
\ 



the Evaluation Unit. Amount and content of information will vary 

depending upon Agency size. 

Within the Agency organization, thjp head of Evaluation should be 

an Administrator. Ttjis will allow input and coordination between the 

Unit and theTrestof the Agency staff. Communication can be enhanced 

since an Administrator will ,hgye input into Agency decisions and policy, 

* l 
which will result in better evaluation/ visibility. 

In keeping with the above staten/ents ^ our durability has increased 

'since we prqviiie the follo**ing type^ of evaluations and reports within 

our Agency:\ % 

\ 

« 1. CoMuct a comprehensive annual evaluation of program effectiveness 
in achieving the service goals and priorities of the Agency, 
pxamples are as follows: 

A. ' Conduct follow-up services, of previous* VR clients. 

i 

,B. -Prepare and disseminate a profile analysis report on a 

I 4 * ' * ' ^ 

'statewide atfd district office level to measure- a goaf~~ 
\ achievement. m 

* / # - 

A C. Conduct case review studies on specific target operations. 

' — ! * ' . 

* 2. Develop annual plans wh'^Lch outline the major evaluations and studie 

: < ■ % • | - ' * 

to be undertaken, conducted, -br completed in the somingf year. 

. \ ' 

Example^ of this a're:. _ * 

A. Develop an evaluation plan for special projects fop other 



B. 



-3. 



sections and the fiefcl offices as requested. j 
Develop an evaluation plan for case review process and t^bjLli- 

! 

zatiori of Case 'Review Evaluatoj>s. 



To respond *to informational requests from central olff ice/field 
office staff "and other "federal Estate, and local agencie^. 
To maintain close consultant' relationships with all administrative 
sections, especially those engaged m client services. 



5. To forecast resource needs,' survey needs for Iphort 
program objectives. 

■ r ' * 1 

As a resilt of Evaluation staff experienced in ev. 



and long range 



^uation techniques 
and Agency operation, we wete requested to do special /stud i as that did 
impact on Agency policy and planning. Some of those Studies and thewv 

v. 

impact are lis|ted below :> V 7 

1. Determine resources^ utilized in college arjjd t^ade school training\ 
that/offset Agency 'costs. ' ^ 

The 'evaluation technique used was the c^se* Review prqbess and 

the findings wef^ given -to >he Agency Director and fhe Executive 

? * * 1 ^ . \ - * \ • / 

State Board. 

Our study provided meaningful informa-tion to policy makers to 



reinforce the decision to triftdify allowable cost^ in training 

* I 

clients. 

2. Determine if existing agency policy on homemaker^ is understood 
and implemented by the field- pef t ^ineJL v ^ 

The evaluation technique used was (the case \*evi^wj process and the 
' findings were given to the Planning Branch. 

Our review resulted, in modification of existing •procedures; of 
providing homempiker*- services to- clients. 

3. Determine utilization of similar belief it s\ 



'In carrying oat this assignment, we worked closely with other 



-4> 



sections in central office aiid field personnels We started a^ 
pilot study in one .district/ office and developed procedures for 
♦i^t#li2dtion« of Similar benefits. 



The pilot study resulted in the 'development of -Agency procedures 
^ and" for statewide implementation after the pilot study is 
complete^ . \ ^ j . 

**. • Developed many reports 4hat are used in decision making such as 

/ ] % ' - 1 

severely disabled repo^-j ; goal^r^port, key caseload report and 

many special reports. • 

The above are some examples /that we feel have increased our durability. 

In summary* I wish to emphasize the following points. 

.1. An Evaluation Section/* s/ durability J in our experience, is primarily 
fa t . 1 / A * - 1 . - ' 

dependent upon its utilitarian value within the Agency not just 

position oh the' organizational chqrt. 

2, Evaluation s£aff sKailld know evaluation techniques, agency, oper- 

' ation and field case /service procedure. 



S. v The Evaluation head /should have an administrative position in the 



Agency. 

It is felt that the following skilljfe aniTcapabilitieis for staff are', 
rl&cessity: * 

* 1. Skills and capabilities of staff fl to do evaluations. 

' .* ' " ; / 1 

2; m Encourage the utilization of /iiewf Strategies and methods. 
£ * ' ♦ - * ' / I \ \ 

3. Develop research of new measm?es { of impact on client needs to 



m include qualitative^ prospe^mve^iarid judgments of clients and 



counselors/ ^ 
* **. Develop measures oi 
programs . 



tcome that wo 



ld^allow comparison across 



/ \ 



X. CONCISIONS. .. 
Suggestions to facildtate the process of establishing the Model 

Evaluation .Unit . • \ > \ 

\ \ ' S - 

A. € The tfrganizatiorial -position of the ^valuation Unit is not as 

important as the Evaluation Unit's ability to provide* meaningful 

information to the appropriate people. \ * ' ^" % / 

■ . . . " • \ ' • / / 

Ideally, the t^ad of the Evaluation jUfiit should be on a'h adminis- 
trative level position that would result in dommunicaftion w£th ~ 
JL other administrators' and involvement in Uecisi^m making processes. 

•\ Define" goals and specifically, define objectives. V * \ 

\ " . : ' "\ \ 

D. Tfie Evaluation Unit should be made up of the following posdtipns: 

\ - ^* ' V s 

1; a statistician who has advanbed knowledge and sophistication 

in analysis; 2) a management analyst ^ho has the capabilities I of 

» V. 

working with other staff and is familiar "with EDP; and 3) a program 

• * ' . * . ' ( v - 

analyst who has a background ' in field operations. The number of \ 
^ tHtese people would depend upon tj\e activities of the Evaluation Unit.\ 
Communication networks and work relationships which must be dev/eioped. * 



r 

en- 1 



A.- There should' be 'levels of communication established. be^w^en- the 
sections in Central Office and the /Evaluation Unit, 

• . jf / - ■ v - 

F. , Working relationships should include such activities as disseminata 
of appropriate information, % input and utilization -as a resource. 

C. Through* conducting needs assessments the MEU should be able to 

determine specifically how each section ..within 1^ie Agency could 

* - " *> 

benefit frqm the information -provided to them. % \ 

#•> • • • » 

D. If the head of the section; is ^in an administrative capacity, he 

} br £he Would have access to administrative meetipg^that could be 



u^^iaed to -describe evaluation "functions. 



Types of information which' the Model Evaluation" Unit sthould attempt to 



provide. • . o 1 • . 

A. • Types of iriformatibn that we should provide T believe ace defined* " 

, ^ ' v . * * & ; 

* ' in our objectives. More specif icaiiy, ^e bo6ld state rn this 
section examples of our obiectivesii for example?: to develop 
annual -plans for.major Evaluations' and studies.- ' • 

qfe Bi » 0th^2? specific examples could pertain to our objectives 1 such as 1 
similar bepefits reporting.- ! , # 1 

*C. Another example wouTd be tne arinuall plan " for case review evaluations 

* , " $ I j * w ' 

4. /Sources of funding. 9 4 / " ' ' \* ' 1 * 



A. No response* * S-| ^ 

5. Da/ta processing requirements for the jjfoni*t J to be responsive. 

~"ThV-Program Evaluation Section shfcuacl have, access to all data files. 

These files should include: client's statistical record (RSA-300), client's 

. ' ' "//'.* " 

fiscal record, and vendor file. / \ - s 

*»''/./ • # 

The Program Evaluation Section should also have access to* computer 

" " ' ' . . - / j V * f * \ 

software capabilities in order to generate data from th£ above* files. In 

Pennsylvania we have access^ to two cpmputer "software 3 packages to retrieve 

^ " ' ~ — ~» * 

data. These are Pansophic-Easytrieve,* which is a versatile, easy-to-use 



information retrieval and data management system, and the Statistical 

*Knalysis System (SAS), which enables us to statistically analyze existing 

' ' . x ' ■ ) . < I . 

^data. fc We also^have "direct access to an online, computer terminal from wjxich 



we can get current client, cpunseldr , 'and district information. In 

' 6 

Addition, we also have regular computer programmed^ reports . which we receive 
on a monthly, quarterly, annual^ or ad^hoc basis. t % 



V 
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working relationships as a factor 

IN THE pURABILilW OF PROGRAM EVALUATION 
\ I » Dentld E. Bossier 
Pennsyiyanla Bureau of Vocational Rehabilitation 
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One of the keys tothe durability of an organizational 
unit is it*, ability to relate to other .units of the 
organization jn a successful manner. The Pennsylvania 
jnodel evaluation unit (I*A MEU) has expanded its rela- 
tionships with other central, regional anc|' district of- 
fice units throughout the State since 'the .inception of 
the contract. A major role of the Program Evaluation 
Section prior to the awarding of the contract was sup- 
port activities to the entire Bureau. Contract work has 
heightened this involvement. Examples of such on- 
going relationships are: * ** - 

1. Case Review Proces s • This requires inftut from 
literally -all segments of e a ^organizational unit of the 
Bureau. This input assists in the determination of sub- 
jects for case reviews, e.g., homemaker* closures, 
suitability ^of choice of vocational ^objective, alcoholic 
case i$ views, etc Additionally, top management's 
utilization of review can>result in pplicy modification 
and can create an inter-active environment* , 

2. Federal Standards • Compilation and analysis of 
statewide data for this purpose requires cooperation 
and consultation from the district offices anp* manage* 
ment at ail levels. * * 

8. Facilities Information System (FIS) • This work ac- % 
Uvity involves relationships with tHe Facilities & 
Grants Management Section in order to develop -and 
implement a system. * . ' 

4. Technical Assistance Center (T.A.C.I - The coljec- 
tion of journals, periodicals, books and other literature 
has enhanced our capability to provide sound informa- 
tion to Bureau staff. In just oyer one year, almost 200 
requests for information were filled. "Requests are, 
received from all 15 district offices and all sections of ' 
the central office. The range of requests is for informa- 
tion on everything from disabilities ' and" program 
evaluation, to management, cdunseling, and job place- 
ment.' *■ »„..'•. ' J 



,A specific example; of Pennsylvania's program 
"evaluation supportive role in our agency is recent 
work on an evaluation plan for the newly emerging 
Client Assistance Project (CAP). , 9 * 



s The Program Evaluation Section (PES) first heard of 
the existence of CAPs through a "Rehab Brier 
disseminated by the University of Florida's Rehabilita- 
tioVi Research Institute on November i6, 1979. As a 
result, the PES contacted, the Rehabilitation Services* 
Administration in Washington, DC, to .access a listing 
of operating CAPs. Phofie contact was initiated with { 
15-20 CAPs asking for their annual reports and forms ' 
used. The responses flowed into the PES and were log- 
ged into Ihe Technical Assistance Center for future - 
use.. 



In April of 1980 the Grants and Facilities Manage- 
ment Section (GAFMS) began gathering information to 
request Federal funds for Establishment of a PA CAP, 
with a pilot study in 2 dis>r1ct offices and one office for 
the visually handicapped. Information was 
disseminated from the T.A.C. to G AFMS in support of 
thcproposal. j , 

The PES assisted in the development of the ^valua- 
tion plan with presenUtioryrto district; regional, and 
central offices, and to the cKp staff. The purposes of 
the meeting were; (1) to develop an awareness and 
understanding of the role ^participants in the projects 
(2) to, increase understanding of the evaluation plan 
system and purpq*e;(^ to initiate a uniform procedure 
at the onset with an* awareness that modifications or 
deletions could be instituted when necessary; and (4) to 
elicit input for any preliminary revisions to. the evalua- 
tion plan before it bee 
1981. 



perational on February 1, 



Topical areas presented by the PES at the meetings 
revolved around a / general overview of program 
evaluation, discussion of GAP program objectives and 
how they are measured, review of ail forms to be utilize 
ed and their flow/ Var/ous audio-visual techniques 
were used to promote discussion and comment. ' ^' 

, Further PES Activity "in the CAP will involve 
monitoring of progress, collecting data, measuring out- 
comes, analyzing/ data, and .dissemmaton. Frequent 
.PES meetings are planned with Bureau field, central 
office and C£P staff 

A nigh priority ,of the PES, in support of Bureau ac- 
tivities', is'timei:;and rwiable assistance. The Federal 
contract has allc wed theVPES to enhance its ability to 
provide a variety of activities in a meaningful manner 
to the Bureau oftyocationa] Rehabiltation. 



"a 



If. you desire/ more ^information on this topic, 
telephone the writer -at (717 )y 87:5123. 
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COMMONWEALTH OF PENNSYLVANIA 



April 25, 1980 



TOt 



Dr.' Jenkins * 



7 



Doh'.Hossler fft 
from: Dick Cohen 

.Enclosed is* the information compiled and updated/to comply yith the 
tequest for operational tracking systems to facilitate the development 
of a> conceptual framework due-/ on May 31, 1980. 



The contents enclosed are as follows: 



1. Original memoranda to initiate development of OT^- \ / 



2. Index 

3. Completed 0TS*s 

4. Extra Blanks 

Enclosures / 

cc: Hr. Guise • ' 
Ms. Worley 



r 



A.V 
* 
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COMMONWEALTH OF PENNSYLVANIA 

January 28, 1980 



TO: 



subjects Operational Tracking System (OTS) 



All Project Staff (Evaluation and Facilities &< Grants) 



FMOM: 



Harry ty^/fcaj^e, Administrator of Evaluation Unit / 
Per: Diil?"cohen/Do1^ossrer ' * / 

As you are probably aware, during the contract organizational and planning meetings in 
November 1 1979, we were assigned the responsibility for the Operational Tracking System (QTS) 

that will accompany the conceptual "framework deliverable due in May of 1980* / 

* - • • / 

Many of you may have already established an OTS .for your assigned projects or at least ,have 
„ formulated a visualization of the sequence o£ key activities and dates related to your? 
project. r m p 

Since we are now four months intq £he second contract year, it appears appropriate that an 
OTS for each project be formally established and detailed to the extent possible. Cpmple^ 
tion of an OTS for each project will also serve to standardize the OTS format, coordinate, 
contract activities and facilitate the preparation of the Hay 1980 deliverable (revised con* 
ceptu£l framework and operational tracking -§ystem. ) You can expect to be contacted in April 
i960 for an updating prior* to submission of the OTS in May. 
' * * 

£ach task coordinator will have primary responsibility for planning and managing each projects 
within hii task. Attached herewith is «a separate OTS^orm for each project assigned to you. 
, Contract deliverables have been entered. Also entered are other estimated product completion 
-dates (Task 2 and- 6) as established by the November 1979. planning and organization meeting* 
Mr. Guise has reviewed the assignments requiring tracking systems and some changes, deletions 
and/or additions may h^ye been* made. All projects under six have been included, even though 
they may have a low priority, since it is anticipated that work will be started on these 
projects sometime this year* # ft 

To assist in standardizing the entries, a completed sample OTS is provided* 

'JPleasfl return your project OTS to Don Hossler, Room 1318, no later than February&25, 1980. 



OPERATIONAL TRACKING SYSTEMS - 1-28-80 

Each task cgordinator will be responsible for planning and managing of each project within 
his task* 



TASK COORDINATORS : , 1 - Jenkins; 2 Chopak; 3 - Jenkins; 4 - 
7 - Dick & Don; 8 - Jenkins & Guise; 9 - Jenkins; 10 - Guise' 

The following projects should have .an 0£TS completed and a cot/y 
t February 25, 1980: J - * 
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Activities not-been pursued further - 
.than an initial meeting in March. 
I980v This' activity was not origin- 
ally contracted in first year and. • 
will require a substantial' commit- 
ment. Further direction is neces- 
sary from^the Federal level for PA 
.to initiate specific work agreements 
and staff commitment, , - 
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THOUGHTS ON 
FOR 



DEVELOPMENT OF A CONCEPTUAL FRAMEWORK 
A PROQRAM EVALUATION SECTION • . . 



The specifications for this contract required the development of a 

... . - i 

"Model" Program 'Evaluation and .Management Information Support Unit. -In 

\ m v. t 

-this context, the, ternl "Model" would appear to have at least, two levels 




of meaning A A moclei might ibe considered *to be an ideal or exemplary 

■ ' y- . ■ h/ . • . , - 

method o^ functioning. 1 "This was clearly one aspect of the Rehabilitation 



Services Administration 
xLes that "given^su 



s intentions since the units were to serve as' 



: ficient resources, State ^Agencies, regardless of 
i) can- develop effective evaluation capacity" (OHDS, 1978, P. 6), * 
Howe 

which is to serve as "a 



ver, a model also h4s a theoretical or scientific leveJL^of meaning 



representation -of the underlying structure of 



ght be conceptual., ideal, or real. In 



under, study, a Hiodel can provide a 
shiis" (GAO, 1978X Brodbeck also 



a(pro<2ess or system. 'The system mi 

ZenA&l, a model/has a simple and/or manipula table structure relative to 
the system it represents. By making explicit the implications of alter-, 
natiy e ftslhanigtions re garding key relationships of the -issue or system 



clearer understanding of these relation- 
agrees_*that *a model has the virtiire, of 
beiijg easy to maijiRU^te so tjiat it "may. help the ^discovery of. principles / 
4hich it, works, if these are no A already known" (1968, P. 58Q). This 

thor does / however, argue against.ia casual use of the^cpncept o*f a 

N - - ' ~ ' " * * * ' 1 ' ' * 

Model" describing models as *two theories whose lows are structurally similar 

to each other. Although the .contract's excursion into the ^rea of -model 

building would probably be closer to^the process of creating^ ahr analogy 

the^n model building w^iin this .precise definition*, BrodBeck's isolation 

of the. essential criteria for a model's adequacy is useful for us since 

he describes isomorphism as .a model's essential characteristics. "Isomorphism 

' * *. * " * x • w « 

requires two conditions. • First, there must be a* one *to one* correspondence 



16 
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between the elements of the mQdel aitd the elements of the thing of which 

*> ■ * * * 
„ _ p ~ - ^ . ^ * 

it *is a. model... 'Second, certain relationships are preserved 11 (1968, P,. 580) 
The quality of isomorphism or correspondence between the model and that 

which it seeks to describe .becomes a test of adequacy of the researcher's 

* 

efforts. Much of our eftfbrt has, been directed toward creating* an 

* *** ^ 

A ' \ 

adequate description or our "Model 11 of the actual operations of an on-going 
evaluation unit. . 4 ^r' 

' * In an*attempt to describe more adequtely the criteria whieh.'are, * 
necessary to evaluate the adequacy, of a,* model, we find that the work of 
DAvis ahd S^lasin is useful. Building on the work of several other authors, 
they suggest the following criteria : t « ' 

O i 

A. v The model, above all should be practical.' 

B. The parts of the model shoulcl be manipuiable. " J * 
C'. Economy of u§e should* be a primary consideration* ^ , 



>ortant . 

_ai4 

i 

E. The model should be comprehensive. 



D. Ease of communication is important. ■ » 



F. .Synergism - "the forces of factors working together - is important*^ 

to 'Consider." t \ ' . * * * * 

» - " 

G. The model should l¥nd" itself to intervening iiu phases. ] * 5 

H. „Dif ferential iiJvestment in ^working >with the components of the model, 

r * i * 

> should be possible. * . r» 

- * • o " 

* 0 /-►- 

I. The model should call attention to how the change process influ- 

» 

ences the rest of the* system. * n 

* . ~" * ' ' * ' 

J? The model should be flexible and versatile enough to apply t?jf 

. • * . . ' r - c . . ; * 

different organizational systems. •** , , * . 

The' model should provide a ba,sis for a subsequent evaluation of' - 
the effectiveness of change . ^ , - . . " 

L. The model should recognize the humanness of the participants. v> 
involved. (1975, P. 634).- ... ' ; 



« * This list has served to provide both criteria for' a test of the model's^ 

< . \ ** * 

v . adequacy as well as to provide general principl/es for, its development. 

Generally, .the process of model bu^diflfc^is recursive^ so that the*e is 

• constant feedback* into the model as it 'is /.evaluated and a dynamic quality 

^ is thus attained insuring that'th'e model will b^constantly refined and 
/ • / 

made more adequate. or isomorphic. * - • 

As-Wildavsky points o\ir ! l An advantage of formal analysis..^, if that 
l- 4 it does n6t depetid 'entirely on learning from- experience*,., by creating 

' model abstracting relationships from the areatg^of/ the universe they wish 

- ' n \ : y * * ' » ' 

"to control , \evaluators^Sfiric to' substitute manipulation .of their model for 
efforts Iti the. world. By' reflecting alternatives kheir ^models tell them 
* will -work out badly (or not as \i|jj^\as QthersT^ these' analysts, save scarce 
^resources 'and protect the public a^ilst le^jwor^thy actions" (1978, v P. 92). 
We flow .recognize model constructiqn and model description as an essential 
portion of "the contract's work! . The RFP contributed to the process., of, 
model articulation* by specifically requifriog a continuing Series of 



deliverables «on the conceptual framework and evaluation plan. In doing so, 
it provided both a focus for many of the, activities as well as setting 



parameters for them and has facilitated the continuing development of 
-the model. s . • 

CONCEPTS OF PROGRAM EVAL UATION ' - "~ 

• < i ■ ■ > v 

• " . * T^e Rehabilitation Services Administration suggests that "the process - 
- of^ measuring, analyzing, and .reporting program accomplishment Relative 
t<s> the program goals f i*s .evaluation." Others' conceive of evaluation in a 
somewhat broader sense suggesting that "the evaluation of a rehabilitation 
program seeks to determine the effectiveness of the program*ifi' term's of its 
predetermined objectives."' Both concepts indicate a clear focus on the 
«Ty identification of goals, or objectives against which the program can be 

. • .18 . f _ _ . 
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measured. This may include such .questions as whether a program 'meelts its. 
objectives at all, to what degree the objectives are <met, and how effici- * 
ently this is conducted across immediate, intermediate and ultimate points. 
It is also apparent that no.t only- rtfust objectives ie identified, but that 

- ' ' ^ o 

a relative value needs to be assigned to each in-order' to make the most 
effective' determination of goal attainment. 0 '* 

The 10th Institute on Rehabilitation Services (Baptista, 1972) out- 
lined a systematic plan for program evaluation as: 

A. To determine agency goals. * 

B. To determine the effectiveness of agency service delivery.- 

C. To determine vocational 'rehabilitation program needs. 
~"~'d. Tq^ insure efficiency in program management. 

^In addition to providing an initial statement on the purpose and- 

' *- } 

conceptual orientation to program evaluators in a practical gpd straight- 
forward format, the documents of. this Institute also gave a basic outline 

for conducting local evaluations which included: - , 

X * 

.A. .Spell 'out objectives for , the program being^valuate'd 

— — S e le ct obj e ctiv e s to be o valu a&ed and determine statistical 

•' '.,.>» * 

( analysis. * ' t 

C. Construct; (or select) evaluation instruments. ' 

Select samples. v * < v * 

v. ' 
E. Determine the, points at which testing is to'take place. 

\F. Conduct statistical analysis. / . " 

. Q. ^Develop conclusions and recommendations. 
>. . * ' 
Proceeding from -the identification of prpgram goals and objectives ' * 

there are several d^aSions that need to be made before an evaluation pro- 
gram" "can be undertaken. These decisions determine the process, scope and 
methodology of th^e valuation process. Evalul^ion may be either ' / 



conceptual or empirical .\ * "Conceptual evaluation is a process of logically 
or rationally, analyzing a problem area and determining the parameters,, 
^structure, or implications at an abstract or theoretical level in order 
to provide Information to managers. Empirical evaluation involves the' 
collection of information or data usual^ through observations or special 
studies within an area in order to gather^ i^ormat ion which bears on - 
the issues at hand. , ~ 

After determination' is made as to the nature of information 'required 

V 

for a program in s terms of its conceptual' or empirical base, consideration 
needs to be undertaken as to whether the assessment will occur at the 
contextual or experimental level. Contextual evaluation models holistically 
examine particular' program operations and attempt to 'explainyhow a program 
has developed, what it^'do^s, and how It can be altered- It focuses on 
action "programs which i/r treats as ongdlng social realities by directly * „ 
studying their everyday activities, particularly as outcomes emerge, from 
continuing interactions among program actors . Experimental* approaches tend 

tcr deal with programs in terms of a generalizable analysis leading to 

\ "V 

cause and effect statements . " These tend to focus on* discreet and °isola table 

fagtors within the program in an attempt to show that direct relationship ' 

' * ' / J I 1 "' ' ' 

'between input process 'and outcome. * ; 

* • • <; " " * " ■ V 

The purpose of evaluation also jieed^ to be considfered in terms of . 

the utilization , of the evaluation' s. product^. .Program evaluations m^y be - 

either fcummative or formative. A, summit ive evaluation. attempts 'to' provide 

direct statements regarding whether a program works or doe§?,.not work. It m 

looks essentially at the overall outcome or impact of a prograttf and attempts 

to evaluate the tota*l merits or worth pf, it s* products / The process of 

formative evaluation is concerned with identification and description of, 

the processes within a program and the analysis of ways in, which the 

• l ' 20 . 161 ' 



program can be improved or the functions involved in the program described 
in a more comprehensive manner. Formative evaluations Usually seek to 
,imprx>ve the'ge'neral effectiveness of ^ program rather than to make 
decisions regarding cpntimfeiion o.r termination as is the case ift the 

/ 



summativ'e* evaluation. 
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SUBJECT: Evotuotcon Section BVR', Organization, Function* a 
and Alignment Handbook*- Sta^ Responsibilities 

TO:. A£t Eva£ua^con Section S&i^ 

FROM: Katttjr W. Gac6e, AcfministAaton. - 
Evaluation Section 



\ 



The Evaluate*' Secti&k, Organization, functions, and Assignment 
" ' 

Handbook is a reference manual having #ie p/umaAtf purposes otf providing a 
man^ema*it £oo£ ({oa. £he A<fainist/Laloti ojj -the Evaluation Station and, a* 
needed, ({or i/ie Evaiaatcon ^^tion 4&x^. The handbook aLso has important 
application in training nexo section sta&&: 

c 

* * 

The coivtents Include organization diagram and describe ZnteA- 

* - * ? * 

reZatlonships^jetoe&x tuition sta^ and other Bureau activities , section 
policies and procedures and scheduLes otf recuAJung^ pro ducts and ixeio- pro jacts . 

7 'Since recurring products and new> project activity in the ti&Uudtflon 

Station is 6u.bje.ct to continuous change, the unitization o& this handbook . op 

K ■ • ' • ' * , ' ' / * 

can only be elective itf Lt*is maintained and updated in an acaytate and *f * 

timely manner. 

V 

To simplify the task o£ maintenance and updating, tixe handbook icill 
be established in one cojxj and located- in the ofi&ice o^MiZ Administrator 6^ 
the Evaluation Station. 



In order tQ achieve the .objective o^ the \w.naboe\i\reilectii\g', at 



« amj'giv&t tlme^Jixe EvaZuaJUon ^Section owCputj -^qJU£'£tcjfyn sta^i m6*t share 
in^utf cjoiibuuioas malntemnce and updating 6& *'tlie handbook . % 




I 4M 



T/ic fattaouig odtbbiu,piocuiuA<!A to- be ioUomd by all 6tnU; , 

-a . - > 

. •• Pon /fo-i^cA l& signed the AefpoiuibiUty oi oyeAalt maintenance, oi the 

■ handbook and a^jipdaUna in^o motion i* provided ■ tSJujn . Leah Kuhni & ' 
designated ai> an alternate. " 

. WiAUri indJviduatlcj signed AecuAAing pAodacU and. new project Jiupoml- 
biUtiet, each 6taU membeA ti Ad>pon6ih'le to pAovide-Von HouleA ok Leah 

. Muu> toUiv&ntly infamatwh tonceAnXng 6ub6taivtiaiycAange^\r-'SpeU^icaUy, 
idien tlie natuAt oi AecuAAing product* change and^hen new pAojecU aAe, ' '. • 
initiated, and u.hen they one completed. •• *"' 

» • ' - - - ' 

. To iusrfiivi uiuAe timely and'accuAate handbook maintenance, onle montlity, 
Von HouZoa oyiealx Kuhu ^contact 'each membeA ^zvlm^dact \ 



. Ml^tati ihouU be^maxe o£ the ,oAga^zaUdAand contention the. handbook 
■ . and contribute any bniomaUan (mdUpptopAixte r^en-ial^0^d' fiance 
<cU pc$po6i and dzvQjtppmZKt. ■ ■ " - - 
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iSARY Of LVM.UATWU rEKMS 

— • V ^ k - , 



- MARZNCSS: Activities o & iixt evaluation sta^ .tlicut ojiz conducted &qk ti\c puhpose* 
( - o£ adiievinc\ ttsc/i acceptance and an ui\dcAStanding o& the gaaZA .'and objectives o£ 
pnognam evaluation o/i ipcci&ic evaluation p/iojccts. -r 



Synonymous Tcw\ti\: TnomotionaJL Activiiit^t^i^t^^^ 





JxkcifruAL jEqUSMXr 

- - ; and .&J®bp8%$^ 

■ • _ . m d objectives. . ^ 

^ V^^^I^^ <#ie necc6Aa^y"*gacda>ice to *|&&S tiicut asbuxzs a 

~ rrse ^3^ ^£^3?^^ plamxing efootut and is revised as needed. * 

r -'^ ^~ - Synonymous T.e*m.U):.. Concept Visualization, Planning G.uidance ' ^ - ~~' 

r ~ ~~ PATA: Vcuta foims */ie banc e£emert& .o^infamation/expA^se^l in facts, number, 
" tettete ofi symbols- and ubua^y^^n"be peaces SZdTSy a compote*. Vcuta is tlte 4 



huJ&aZ/MAM project fallowing compilation and prion to 

.ancilysjjy^xxt^c^ . Vcuta is ' processed into infafimation.*' 

-Synonymous Te/un (* I * Rawr Vata t - Statistics , bompyut&A Vcuta 



PlSSEHIiVAnOJ/J. Wphase oi the eydZaat&*i JP^^^^^^^^^^^^SSSE3^^^iz.^ 
dcXcAMnaXion o '& ptedv^^v&zftS^^^ 

tAatjsmitJ^ j0&fru&hr.~ ViA- 

t^^lSst^^ -t(ie needs detetowiatipn 

~" f^Mz^f^ther zKm^SMon process cycle. v ^ 

" Synonymous Teim[&$*- 



WCUMENfATftW: flatten material usually 2h tiie fansf 6 i' records and reports pro- 
vided hif an awtSxbxitatXv^ resource mat 4ufo r ta)itiates f recorded factual data on. 
analytical^ conclusions. / 



instrument developnoit 



Synonipnous Tc/un{s] s Ja^^^ca^cok MateA^at, P/iooj{ 
EVALUATION ftiSTRUllEUf: Tkc docwn&xt established cfatutg t!ie\ 

phase c& &z Evaluation process cycle. It iA utilized durcing ^fc implementation 
phase oi die %valaaXJjon ajc£t md facUlttatte tl*c collccZioii oQdcuta,*" 

' \ / # 



* 4 



C<wc Rev/>£w Tom, .tvahiatJion Tonm, FolzMiip Questionable, ' 
EPP fa-ut*ou£, etc.- ' \ ^ " % 



feci 



0 



39 



184 



- V 




o^ic^d m^4*on,--.Tfte .UtabUilvnvU oi goad oac adiuSk 



Godfcs p10v4.de. 



<eutetf, Vcufa 



4. _ 



. J5SZ VR£££^ ^ and 



\ 



Synonymous TzAmUi: 




\ • 



/ > 



\ 



183 - 





!||§^ activitieA to 

^, ^%a^V n rf o&jettive^ oi jfa'waUm^ ttonUXKUig m 

^^^Uc^mdij incZude jxctiote to' modify ox coaacU previously ptatmed pnocedwtlb 
maintain coniiAtent goal and objective. dixectcon. 

Synonymous 'TeAmUJ: SupeAv-Ue Manage, ToUofi Up - . 

WEEPS OETEWIlWATWW^Weeds Vv&minaUon lt> tixe iitei 6tep in the qfct* oi the. 
evaluation pwccw; tixe agenaj'* lifacUvVMA* -or t/ie aduewemeitt o j " 

aoaJU and objicAivte^M <a> tin bati&Uo*. e^tabLi^^jibxaXion pAOg&l. Tae 
Weed* VUtmoxa^m^^^^^ a zooxdinaUct e££o&±nyo^ 



§&2^£5§ii^ ^ ~evo22fa£ion project. 



^Synonymous TftftmU): Heeds Sm-oig, Weeds A4<$e6<smertt 



OBJECTIVES: Objective* axe xetatedand saboxdirtate to agency goa&. They axe a 
moxe t&U&ic ttatemcnt o&~riiat'mu*t be aacompttAfeed <£a achieve. tlie~agcncij goats, 
Wie oh Active 6tatamni may also detalL hem and when the. objective. & to be. 
'accoxzZishzdL ; ■* . 

' i ' " * * • _ ' ■ 

PROPtlCn. A Pxoduct is the. j^afi§documen£ed xeAuZt& o&m evaluation pA.oje.ct. lis 
conicitts. ^ a* com£c£atam data, A% data analysis into in^Ajmtion and 4e- * , 
comn&tdations &ox utilization* * " 

Synonymous Team fob- BvatuGtidn P^o/ect Kepo^tt, Veliyefoble. 




PROGWi EVALUATION VE).: A geucAat -tlto -t/iat define* a£X ^/ie ac^c^rCt4^4 ttiaZ 
otMiccci^aAy cQfflpoue»t<a o(J ^e evaCuktcon p^u?ceA4. I-C ajcZicaZ -ox na,tdn.<L and 
nojnatttj ^taxts .witli needs deteAn\bicLt)Lon and pA.oce.td6 tSiKoaqh UjittAwnoit develop- 
ment and .design, £itLd tteting, ^nptdixentjaXlon^ compilation and convey ion o& data. 
m ntq in&o&raUon and AtdSmmpidittLote A publication 0& a product, dUt>cniairition\cuid % 
.tctizatCpn acXiv^tiei ajtd* tvcLti&tiow. Paoqajwi Evaluation iA an ilnpojttAnt cdm- 
)one)tt\ofc a management infaoAmation Itktan. • * ' 



it 



a;;. 





• V. 



•V 



J ■.. 




v3 ^^*^E s rK^ V-?~^ 



^ S^^-^ ^ totd'vSio will 



** 

UTItTZATTO* 1 /: A pfiaie o£ £'te ftca^aatiot p*ocei4 q/a£e #u#: is conccAnecf w-ct/i ,£/ie 
cooidinatzd activities o& Xiiz o.v,aZaa,tibn project 6£a&& aid project usqas. T/ic 
* .paApo^c oj{ utilization activities is to os6uaz*usza aivawnos* , und<yis landing and 
agi<LQJKe.n>t i)\ £hz puipoti aj\d use o& evaluation project matoAiai. iitHizaticru 
activities asit ctoitltj itlatzd and usualty planned with cUs^mination activities. 

Synonymous TeAmU):" U6za Suppont, ttoe. 



-J 



iERic; 



42 



-isrf 
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^^^^h^^^^^^m^^mmoN system „ 



- - ^V- ' ^j|gpf^ - 

- weeps determination 



EVALUATION PROJECT 



\ 



INSTRUMENT DEVELOPMENT 
.-EVALUATION INSTRUMENT 







IMPLEMENTATION 
INFORMATION 



PRODUCT 



DISSEMINATION 



UTILIZATION 



DOCUMENTATION 



capacity mnvw\ ■ . 



MONITOR. 



9t 



AWARENESS 



TRACKING SYSTEM 

/ 



- 4 



fi kjc 



oU3 






IHFonrMTioM 

SYSTEMS .4.-3*^3 



OTHER. 



r 



PFfoGBAM 



EVALUATION 
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EVALU^rtorT 
'PROJECTS 
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UTILIZATION ^LAN FOR CASE. SERVICE EVALUATORS ^ APPENDIX 7 

r~ - " In the iftterest^of increased productivity and more «f Cicient management, the- 

^ "dUties^and re^on^ib^lities^' the Case, Service Evalua tors- were further* defined and 

^ an annual plan- for case revie^i^was d&velopsji 



- - - T&<ffi^gtahal evaluator is more dire^tl^nvoT^d-in the* instrument: development • * 
and design phase, of the Case Review Prqce,sa^ Wiifaia ±he~guidanqe .and supervision Vf ^ 

^th,e Administrator of the Evaluation "Slb.tion; , each awaluatdtf will be assigned full " 

-resfgoasibmtK^f or Instrument developnihf-afid tie-sign for a specific: ^aiua-^¥^5e^J 



This varies from ourpast practice of Accomplishing th&T£as£as*a coordifcl&Ld ' effort 
in that each evalua tor was assigned only one ■ s^&ific part" of -the^nstrument ^ ? -;• 



' ^^development and design. % 

This. responsibility vdfll generally r^qyi're \Yt^ 'dewlopnteif^m^^^^ii^of- v , 
:j the following : - ' ' ^* - * / ' 

* . !• \ statement 'bflhe .objecfeve oT- the ^speaifi crease fevieyir ^ " T*" - % 

" 2. -*Case review question form , ,° * , " -^^-E^jSS^r' * 



3. Question interpretation^ \. >^ * * 



1 4. Guidelines. ^ ; ^ X?^- 

^ V * . t • — ' l . ■ ^ ; * ^ 

5. The system and format for * the oompllation of results^' 1 - " - 

ll._\^? he following is a suggested list oT steps that may be follpwed in the ' ' " 
perfoitiance of th^se assignments: * , ' , 

% . . !• Assignment of the Case Review Project, by the Adirlinistrator. - \ "'' 

.2. "Within the supervision and guidance of. the Administrator, the development' 

of the statement of* the objective of the case 'review. ' ¥ '* v 

- ' % ' # * > * - ° . • • 

3. Preparation of the ffrst dra'ft of the review questions, question interpret 

% " ' / ' ' - ' '* 

t tation, guidelines ^and results^ format*, - ■ ' • ^ 

4. Presentation of first draft ^to the Administrator and other evaluators-.- a " ' - 

5. Revisions as the result ^of* discussions ana decisions of the previous step, 

• , . t • »6. Pretest of; instrument for reliability by. all evaluStors. \ . " . ' ' r * A • 

7; -final reyisiohs and submission of tha material for the initiatioff-of *the^-' v'J 

' • • ; ,1* \ . v . . '• * - ■ - \1 

: • case review. • '• • ■ „ ■• ; \ * • **• -.^g 

•;• . .' . • • • ^ . • *. " : ■ ■ ■■ - . v 




Such meetfnfs as 
coordinated v/ith the Administrator, hov;evog^^»a^ 

■cation should. be jutdlised. "\ 

** ' ■ - ^ 

r* . - ' 

The 




^»^^^v^l-tt'aiioft^ction staf?"Tn devising questlW^^ToaS' 



raiu^torsf for t-HeiS. -iim»t V* ' ^TZ-.'. 




( _ «w;araraal plan for case^^lews^iBs^^dule specie projects. ...The tentative 

■S^^- ^^^^ ^ d v fini ^ h ' .?e%>&iVr,the.p^C?od for. instrument development and desW^" 



Lown in^the 



- *en&rks ;cc$^iff^t k 4"^'. and, will U~ Tsrtffljfcit*: to evaluation, section operational " ' ,/ 

~ ----,3. . .^•f* >- . — >t ' • /■ • • v - ■• 

•••"-fajt^^Sat occliff^M^tteii&uai^i^.^^ ' ." - •; £>V V""."^ 

* The as signmfents "o^B^rutt^fc --de ; velopia^it^ as Xes<j;9^J£ will"" procede ^ ; tent^lye ' 
. starting dates shown and may .he initiated by- the. evaliiator. as ftime is available . It 

is expected' that "concurrent/activities by the evkjuator wj^ll occur .'so,'. previouslx 
_ available' time -during* and, between reviews; can now. Ve more« ef Fieiently utilized. - k f' 
. -' The instruments for reviews scheduled 'late in the' annual period may .be. worked. 
f . early in the period, and held in various sfeps «(f cteTelopjnani^oi: "later'utfil^ion, * • 

U " ^ • • ftow6 . ve ^^"t xe a'mpbrtant that .|he"eyalua f tor assigned the project'" has ail the material' * 

fv" . t-A •* r : /' « '* " .' .". ' ' '* , r> -V 4 -'-"-- «• ' 

;;.*"''V- : C finat'V^ttstfif^' cte.lfeview? sftarting dam M'.- » 

. f^lan ToAputiliaatioft pf case "sjsrvi^ ^yai^^rV'^o'iSli be used as al 

A WOrV TVfem Jl^v the . iaKrttrn ' k«^i'io1 rxT «^ aai.TO ^^-X^^i*^^ ^ , ,^ 



^ , ; Tha'.Annual 




1 • . Be views JPede rtfL 



,2, Completes reHewt^a 




.^--Wft Conduct coriipWienaiv^ continuous, ja^^ 



3sz 



, v of strength and ^eatoes^in ao^ 



td. patt§3 




■ - . 6. Assists in».cqnipiling alj.. necessary, data - for '-a /anal report oij a-st^te^'ae^^^/. 
HvV t * - Wicf 4ist&ct office level. • \ ." / •" ' : . 

!(./•-». VV V " •- 7.' Conducts 'follow-up in/di'strict. offices' ih:',the ; regio'n. ' • « 

-"i ■ }..?.•-,--," • •' ' :..<' " • ... \. ■ •'. ■• «'* 

- . ' * - av- l - - - ., f * 



'* / f 'I " • 




4& 
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PERFORMANCE GOAL .AND WORK ACHIEVEMENT PLAN 
BUREAU pr VOCATIONAL REHABILITATION 
£ROGR£M EVALUATION SECTKIN , , 



PERFORMANCE GOAL - Identify, develop and implement an* annual plan, Jan. I, 1981 to Dec. 31 
^981 of case reviews & utilization of Regional Case Review Evaluators.' 



WORK STEPS ' J BEGIN 


CONCLUDE ' % 


RESOURCES 


k COST 

1 ANALYSIS • , 


- RESPONSIBLE 

- PERSON 


JL • nSs ignment 
of Cafe ReViet 
by Administra- 
tor- of Evalua- 
tion 


k 

. 8/1/80^ 

1 

4 • 1 ' 


8/31/80* 


■X ■ 


lilgtimated 
Staff Hours 

3 , 


Evajluation Adminis tr a- 
« 


'2. Development 
of Statement 
of Objectives 
for Case 
Review ^ 


. 9/1/80 1 

• 


9/30/80 

* < 

N 


Case Evaluator 
Program Evalu- 
ation Section 


Estimated 
Staff Hours 

25 


Evaluation Administra- 
tor - 

Case Review Evaluator * 
Region IV 

t 


3. Develop * t 
design instru- 
ments' & stan- 
dards, incl. 
' pretest for 
reliability & 
selection of 
appro^ r sample 


n/1780 • 

7 


•/ 

1 

12/31/80 


Evaluation 
Adminis trator 
Pr©g„ tv,alua- 
tion Section 
Case -Review 
Evaluator 


Estimated 
Staff Hours 

. 350 
♦ 

\ 


% Evaluation Administra-j 
tor . j 
Case Review Evaluator \> 
Region jv N V ! 

- • - ' 1 

- . . 1 

" • i 
t 


4 • Implementa- 
tion of Case 
Review, - col- 
lection of . 
data and 

flAdl I/O 1 €J 


VV81 

> 

\ 


3/31/81./ 


Eval. Admini- 
strator, Prog-* 
ram Eval* Sec. 
Case Reyiew 
Evaluator 
Central Office 

— ■ ■ t 


Estimated 

Staff. Hourls 
t 

1;036 


Evaluation Administra- 
tor y ' 

."Case ReView Evaludtor 
Region iv » 
• • 


5. Preparation 
of Case Review 
results, -incl. 
recommenda- 
tions * < 

e / 4 


i .« 

4/1/81 *' 


5/31/81 . 


Eval. Admini- 
strator, : Prog- 
uam Evaluation 
Section 
Evaluator 
Central t Office"' 


Estimated 
Staff Hours* 

* 

. 740 > 


Evaluation Administra- 
tor *• ,< 
Case Review Evaluator 
Region- XV '# 

• 0 

• • • 


6. Dissemina- 
tion, monitor- 
ing anjd feed- 
back 


* • 

6/1/81- .." 

• .■■'■/" ' 


7/31/81 ,r 

* * « 

• « 


Dist. Pffice « 
Reg. pffice 
Central Office 

* * * 

* ■ 


Estimated 

Staff Hours • 
• • 

'ioo 

^ * 


Evaluation Administra- 
tor 

Case Review Evaluator 
Region XV 

• • ! • . - I 





* « * .* x ' 
V**. ' 



:4 

6'- 



192 

48 4 * 
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xESSOSCi^CS GOAL 



PERFORMANCE' GOAL -AND KORK ACHIEVEMENT PLAN . 

3UREAU OF VOCATIONAL REHABILITATION 

PROGRAM -EVALUATION SECTION . 
• *• * 

Identify, develop and implement* an aggual pLan, Jan* 1, 1981 Co Dec* 31, 
198JL of case reviews' & utiliza't^a'tH^ Regional Case Review Evaluatbrs* 



IWRP - QUALITY OF ?ROCEDUJ3E5 USED 



VOSSC STEPS 


3EGIN 


CONCLUDE 


/RESOURCES 


cost • „ • 

.ANALYSE ; 


RESPONSIBft • ? 
PERSON ' . * 


1. Assignment j , , 
Tof Case Review % 
Sv'Adainiscr^-j -3/1/80 
Tor of Evaiuan 

tte _ . j . , 


8/31/80 


* •% 


: 't*^ — 

Estimated. 

Staff Hours " 

. 3 

* if 

. i t 

:y 


Evaluar ion Admini c ft^-n^ 

tor "*~ 


1 of Statement i 
CbjectiTes I 


. 5/1/30 


a. 

9/30/80 


Case Evaluator 
Program Evalu- 
ation Section, 

✓ 


Estimated 
Staff ,! Bburs 

25. 


, t 

Evaluation Administra- 
tor X • 

Case. Review Evaluator 
Regies II- 


2- develop 6 
casign i:^ tru- 
ants a stan- 
dards , incl* 
pretest for 
raliabilitV ^ 
-selection o^f 


i2/i/8a 


V - 

•3/31/81- ' 


Evaluation - 
Administrator 
Prog*. Evalua- 
tion Section 
Case Review 
Evaluator 


Estimated 
Staff Hours 

450, 


Evaluation Adminis^tra-* 
.tor 

gCase Review Evaluator 
Region 


A* Iottiementa^ 
ti'on-of Case , 
Review, coif 
lection of } j 
data and ' 
analysis, • 


4 A/81- 


6/30/at * 
* 


EvajU Admini- 
strator, Prog- 
ram tv&l . Sefc . 
ftase^aview* 
Evaluator ■ ° 
Central Office 


[Estimated 
Staff Hours' ■ 

1,136 . $ 

v t 


Evaluation 'Adraini-stra- 

y cor t . r > ; - 

Case Seview Evaluatoir 
'Region n J.- ~ 

• . 1 • - ' ■ • ; V, 

. if 



of Case Review 
results, incl. 
recommenda- 
tions } 



7/1/81 



8/31/81 



stratbr, Prog- 
ram £valua.^loi^- 
Sectioti 3 s 3 
Evaluator. 5 j, 
Central Office/ 



Es ciqfe ced 
; Staf % Hours 



750 



Evaluacion Administra-4 
Cor ^ \ 

Case Review^&valuator 
Region II J 



'6. 'Dissem|ria^ 
tion^ mdnjitor- 
,ing and feed- 
back . J 



5/1/81 



^list. Office 
Rfcg. Office 
Central Office 



10/31/81 



Estimated 
Staff Hours 

10Q 



Evaluation Administra- 
tor • - j / : 
Case Review Evaluator 
Region n 



I'JttUrOKK&NCE GUAL -AMD *OtlK» ALHLtV&WfcCil' fLAtf 
BUREAU OF VOCATIONAL REHABILITATION 
PROGRAM EVALUATION SECTION - 



PERFORMANCE GOAL 

■■ 7.* / • 



- Identify, develop and implement an annual plan, Jan. 1, 1981 to Dec* 31, 
. 1981 oi case reviews & utilization of Regional Case Review Evaluators. . 

**) 

" \ ' STATUS 14 - STATE OF-*THE ART 1 



1 $ 

WORK STEPS 


BEGIN 


CONCLUDE 


RESOURCES 


COST 
AFALifSIS 


RESPONSIBLE ' ' 
PERSON 


1. Assignment 
of Case Reviev; 
by Administra- 
tor of Evalua- 
tion 

r 

K 


< 

'3/1/80 • 


\ 

8/31/50 

> 




Estimated 
Staff Hours' 

3 


Evaluation Administra- 
tbr v . 

* 

• i 

1 


2. Development 
of Statement 
o£ Objectives 
for Case ■ 
Itey iew 4 

V 


K 

9/i/80 


9/30/80 
♦ 

* 


Case Evaluator 
Program Evalu- 
ation Section 


Estimated 
Staff Hours 


Evaluation Adminis tra- 
ctor 

Case Review Evaluator 
Region IV 

* 


3. Develop & 
design instru- 
ments & stan- 
dards, incl. 
pretest for 
reliability & , 
selection df* 
appr op. sample 


> 


* ^ 

• • 

"6/30781.. 

< z 


Evaluation 
Administrator . 
^rog- Ej-alua^ 
t;ion Secf ion 
Case Review 
gvalua-tor' 

A 


Estimated 
Staff Hours 

350 

1 


Evaluation Administra- 
tor 

Case Review Evaluator * 
Region jy 

• 


4* Im^lementa- 
"tion«x>f Case 
Review, col- ; 
lection of 
data and / * 
analysis r ? 


* i 

. 7/1/81 


9/30/81 
• 


Evalo^Admini-f - 
stracor, fro^ 
ran* Eval.' Set. 
Case Review ; 
Evaluator -« 
Central Office 


Estimated r t 
Staff Hours 

.J, 036 


Evaluation Administra- 
tor 1 . ' 
.Case Review Evaluator 
Region iv ' . 

s 


of Case:. Review 
results ,''iticl. 
recooamenda-- 
tioiuU 


* 

10/1/81 


12/31/81 


Eval. 'Admini- 
strator, Prog- 
ram Evaluation 
Section w . ' 
Evaluator'/ 
Central Office 


Estimated ^ 
Staff Ho«r3' 

' 740 


Evaluation, Administra- 
- tor ' 

Case Review Evaluator 
Jlegion iv 1 


$i/D£ssemina- 
^fori,. monitor- 
Altig arid feed- 

/biclc 

' > ** 


1/1/82. 


> 

, 2/28/82 


•Dist. Office 
jReg. Office 
;Central Office 

* 


Estimated 
Staff Hours 

100 . 


Evaluation Administra- 
tor 

Case Review Evaluator 
Region iv 

• * m 



/ 



\ ! 

:5 I 



► PERFORMANCE GOAL ..AND WQRK ACHIEVEMENT PLAN ' 
, . BUREAU OF VOCATIONAL REHABILITATION 

PROGJIAH EVALUATION SECTION " * 

% ' ' ^ 

PERFORliANCS COAL -/identify, develop- and .implement an, annual plan, Jan. V, 1981 to Dec- 31 f 

. 1981 of. case reviews & utilization of Regional Case Review Evaluators. * \ 

v \ • : • 

SIMILAR BENEFITS UTILIZED 



WORK STEPS 


BEGIN 


CONCLUDE . 


RESOURCES 1 


. COST 
ANALYSI.S 


RESPONSIBLE 
' PERSON * • 


!♦ Assignment 
^of Case Reviev 
by Adniniscra- 
tor of Evalua- 
tion 


; ; — 

'8/1/80 • 


8/31/80 v 


r r 

• * U 

* i 


Estimated • 
Staff Hours 

3 ~, 


Evaluation Administra- 
tor 


Z. Development 
of Statement 
of Objectives' 
for Case 
Review 

r 

* 


9/1/80 


'i 

' 9730/80 


Case Evaluator 
Program ^Evalu- 
ation Section' 

< r • i 


Estimate^ 
Staff .Hours 

-25 * % 

v 


% - ' , 

Evaluation Administra- 
tor 

Case Nlev lew Evaluator 
Region m 

. V . . • 


3* Develop 6 
Resign instru- 
merits & stan- 
dards^ "*ihcl. 
pretest for 
reliability & 
Syelect^on of 


2/1/81. 

-3 «■ 


i 

9/30/81 

r * / 


Evaluation 

Admiriist|at6* 

Prog^ Evfilua- 

tiofi'/Seclion 

CaseJfRejfiew 

Evaluator 

vrf' < • 


Estimated 
Staff jlours 

\ £50 

,\* 

* % 

a 

♦ 


Evaluation Administra- 
tor 

Case Review Evaluator 
Region III' , ; * 

. 5 ''^ * • 

1 ' 


4- Implementa- 
tion of Case 
Review, col- 
lection of 
data and 

analysis • * 

• . * k ' 


- / 

,10/1/81 

* ' 


> % 

'12/31/81' 


Evalu Affmini-^ 
stratonk; Prog— 
ramfEva^tj, Sec. 
Casl .Reyliew 
Bviafuakar 
pe*ra|jpffice 


Estimated , 
Starf Hours 

1,136 

4 


Evaluation Administra- 
tor " • 

.Case Review ~EvaLuatar 
.Region III 

y , > 


$'♦• Preparation 
of Case Review 
x.esults, incl. 
recommenda- • 
tions 


l)l/82. 
> . i_ 


v 2/28/82 J 


Evil ^,4^mini-» 
stiatf^v Prog-1 
raK.^lluation 

E|jaiuator > 
C|nt^|l Office 


Estimated 
Staff Hours J 

* 740 • 
r • 


^Evaluation Administra- 
tor 

Case^ Jleview Evaluator 
-Region^ xil 


6«oDissemina^ 
tion, monitor- 
ing and feed^ 
back^ 


3/1/82 

t 


4/3Q/82; r 

f/l 

• v| 

Mr 


t $st.$Office 
|eg r . .Office 
Central Office 

it 

)■■■. 

t 

i : 

i- 


Estimated 

Staff Hours 
t 

100 

, * t 


Evaluation Administra- 
tor ' 

Case^ Review .Evaluator 
/Region ixj . s . * 
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. ' '•' . :\ , "• * APPENDIX 8' 



• " . \'> •' PROGRAM EVALUATION DEFINITION \ 

\ ' 

'r • » ' - ... 

. Erpgram evaluation is the measurement' of the overall vocational 'rehabilitation 
program: finding out what is working and why. It is the' collection',' measurement," 
and analysis of data regarding the program's 'degree of achievement in a quantitative 
and qualitative manner within the parameters of resources available. 

•••• Program evaluation is an input process involving staff and line personnel at 
all organizational levels. Products developed from the outcomes of", program evalu- ' 
ation activities are disseminated and utilized by' both Internal and external audi- 
ences, interested in vocational rehabilitation. ....... ' ■< 

\ ■ • ! 

' PROGRAM EVALUATION GOALS (WHAT)' AND OBJECTIVES (HOW) 
. ■ r '■ « . ■■■!■. , , 

The ultimate goal 'of program evaluation -is- not just compliance .'with Fedei/al and 
State- requirements, but to. assist in the "improvement of vocational rehabilitation 
services. ' • • . .*.'•„ ■ s 

. WHAT . : \ . • V - ' 

1. To conduct a comprehensive .annual evaluation W .program effectiveness in 
. achieving tlie service goals and priorities established in the Annual' State" Plan 
and Amendments. ' , 

••" \ , . * K 

\ s v how < : ' \, 

... - * . ' s *. : ■ • ■ ■ w '. 

a. Conduct follow-up surveys of previous vocational .rehabilitation clients ! 
t b. Prepare and- distribute annual reports- of the agencv activities. / . . ' 

c. Prepare and disseminate the Profile Analysisjleport .on a ' 3t'a6ewid*e/tfistrict ■ 
Office leve^.to measure goal achievement. 
. d.*. Conduct case- review studies, on' specif ic' ..target operations. •»'" 

• €. prepare reports to Central Office management..* Reports listed ielow'are 

-> examples of Item, e: ) ■ -' .,,*>. ' *j 

' . . ■ 1. State Budget s on Bureau goals* population parameters, -and iinpa'ct^indi'- 
•"' ■ "cators:. y ,^ 

' 2. Monthly, and Annual Rehabilitation 'Goals by region and -district office.?' 
. . 3. . Monthly Key Caseload data report " ..-•'* 

4. ' Monthly Workload Statistics repert' " ' , - ' ".. 

5. Monthly Source of Referral report/ ' ~* 

• f. Monitoring IWRP activities^ via- Annual Comprehensive' Review. - ' r 

> 



WHAT 



* . - - - • i < » 

V. To develop annual plans, which outline' the major evaluations. a»d. studies, to be** 
Y^uno^rtaken, continued., or completed in the coming year. ,7' 

. . • 'HOW- ■■ . • ' • : 1 - ' , 

to. Developed an evaluation plan for the profile Analysis' System.' " ;1 " ^ ' 

> b.^ Assist in the preparation of the Program and Financial .Plan 'of the A-gehcy 
- * for Federal artS State. . '■ * , 



c ' Dev ^J°<P ed an ..evaluation plan for follow-up surveys. ' 

d. Developed an evaluation plan for. current" Federal Program' Evaluation 
Standards. \, 

e. Developed an evaluation plan for case review" process and utilization of 
» Case) Review Evaluators. •* " '. , * 

f . _ Develop' an evaluation ' P lan'~for special projects for other sections and 
. fch^field pf^fices, as requested.'. , 

' ' . ' . '• ''-WHAT ■ ■ ' 

* - • . . ? • v 

To respond to informational "requests from central, o'&f ice/field office staff 
and other federal, state; «&nd lofial agencies. * . - . 



a. 



HOW .• 

Assist all requesters of • 'information in assessing their informational needs 

(2) computer 
ssistance Cente 



aJ .x i.c 4 ucoLCL£> ui'intormacion xn assessing their infoi 
Prepare reqtfest^ for information using (1)' existing reports, ' (2) computer 
software package (Pansophic Easytrieve);, ancf (3) Technical Is 



fc. Measure the' frequency of request, 

f * . * / WHAT 1 * % ' *\ 

To build and refine program evaluation capacityfthat is flexible and open-ended 
Jo integrate new knowledge and techniqttes into the program evaluation system fo 
ou^c agehcy policy v &nd program- development andV^j/aiuat "* 



:ion. 



HOW 



a. . .Develop a pian for a -Similar Benefits .Reporting System to measure types op* 

♦services, sources, 'dollars. saved, and increase utilization of. Similar 
# Benefits. I / • ^. ; \ 

b. Develc^pm^at of program evaluation plans for ofher central- of fic.e "sections. 
- .Examples of completing program evaluation plans are listed below. 

1- ' ' ^^ ent Assista .. nce Project :for Grants and .Facilities Section '* . 

* 2. 0 * Ind&pfendent Liying Project fop Grants- and" Facilities Section 

* 3." SkiMs Training and Enhanced- Placement (STEP) for Systems Development ' < 
t . . Section ' • 1 ■ * . ^ " ■ . - 

4'.* .-WotjArs 1 ' Compensation Referral System fof: Systems Development Section 
c/* Pe^elbp;|». ai^ implementation procedure for a pilot project - for collection 

- of' 5im^| feeifefitsTLn the'Altoon$i District Office, ' • v 
I. Statdwid^Snplemen'tation- of the Similar -/Benefits pilot project. 
'* Peyelob ; %lan for' a Facility Dafcr Base System for central/ regional' and* ■ 

f. ,' DevelopN?^implementat*ion -procedure fo 4 r a' pilot projeetf for .a, Facility Data 

•Base Systfetn;, % v ; * * ** , ' 

g. Deyeloped' ? |nd. implemented. a Program Evaluation Organisational -Handbook - for 
. the program evaluation'*stkf f .. , ^ " 

h. , J)evelopedja* plapf'for the review of .reports generated, by the program* evalu r " • 

ation ; section -as part bf^ the Mkn%em$nt Information System. Thes* "reports 
r were fiel^ tested* with central. off ice personnel, ii the future, we plan to 

4 ^review 'thejsje reports with our field offices* •• • * 

i e 'Developed detailed analysis of the relationship between satisfaction 

ihddces and key service and proces§^>riaicatcfrV .of the Life* Satisfaction "^V* 
Survey's. . 5 * * ; , . \ , p ,^ * ^ • « 



e 



j. Dev&l'opitfg a cross tabulation and Validity checks on the follow-up surveys 
through the use of Statistical Analysis Systems (SAS). SAS is a statistical 
computer softxvare package* / • 

k. ' Utilization of a micro-fiche reader to. extend our capabilities in various 
research ai^eas. * 



WHAT 



V*. ■ 



ilffe 



To jnaintain close consultative relationships with all administrative sections 
within the agency, especially those engaged in client services. In addition, 
relafionships/will be maintained witfh universities, private agencies* and 
Organizations, as well as other state or federal agencies. ^ 

. . , - 

'•.*•*% •- 

a. prepares outcome and 'service goals for inclusion in State plan. , 

b. Prepar^ numerous, activity program reports for RSA and SSA,. 

c. Prepares and provides, technical assistance to the following Administrative 
, sections within the agency. « * • 

!• Systema Deve-lopment Secftion - Workers' Compensation Program, Placement 
^ - Program, Skills Training and Enhanced Placement Program, Peaf and Hard 
of' Hearing Program, CETA and. Public .Welfare Programs, Hental Health ahd 
Mental Retardation Program, Johnstown Center Training Program for' 
Employers, arra we also aid the Psvchological Consultant. • ' • 

-2. Grants and Facilities Section,-' Development of a program- evaluation 

..plan for the Client Assistance Project, assisted in the quest iorinaire • 
design for the collection of" data for the Comprehensive" Rehabilitation 
, v Center Project, developed a program evaluation plan for* c^nte^for the 
. Independent Living Project, and provided the othe^ rehabilitation ' * 
. ^ specialist with various types of requests. *' • . r 

3..* Public Information Section - Write ^informational articles 4 , provide. 

statistical data on our overall program; 'and ^provide .this Section 'with 
information from our 5 Program Evaluation Section Library. t 
4. Training Section - Worked, with this section in providing training * f 
programs an Similar Benefits utilization and the Case Review Prpcess. 
We usfed the Training 3 Section to produce! transparencies for our o'ral • 
presentations at various meetings^ 9 \* * ^ # % 
. ^5... Medical Section Supply statistical data on request, e.g. clients 
with spinal cord jyijuries. *• # 9 

6. , Legal Section - Provides information on the' con^ul^ta^its used by the 
\ . Program Evaluation Section* Also utilized this section for questions 
concerning "Confidentiality of Information" release by pur *agenGy to, 
external audience^ • ^ . - < 4 4 ' * . 

> 1 , x Affirmative ^cfioa Section. - Supply statistical data on request con- 
cerning demographic ch^rteristics sSch as sex,*race, age, and Spanish 
w surname by cdiinties. * • a * 

8. Special Projects Section -Provide statistical ' data on the Veterans 
t Program, No-Fault Insurance Program, Aging Program, and the Drug and 
' Alccfhol Program. ' Completed' a Program Evaluation. Assessment of the ? 
, Aging, Dxug and\Alcphol Programs annually, n. ^ e 



»^if V ^ - :* i * * : 

I^V" 9 # . Budget Section - Assist in the /development of. the RSA-1 Report Program 

»v • Fiaanciale Plan and State' Budget .' Wofk with the Purchasing Officer in 

. . the ordering of. supplies and library books and periodicals fo,r the 

|l«7 4 c , . Program Evaluation Technical Assistance C&nter* 

10. • EDP Section - Maintain a direct linkage with this'section for infor- 
• . national requests that require computer programming. The Program 

1 < Evaluation Section and the EDP Section act as supportive sections to 
* the .central office and field offices. ^ 
d.*" Established good working relationship with the following institutions: 

1. Berkeley Planning Associates 4 

2. West Virginia Research and Training Genter 

3. Qregon Research 4 and Training Center 
• £• Arkansas Research and Training Center » 

*' 5. ^Abt Associates * r 

. * » 6. Walker and Associates v , 

7. National Association of Rehabilitation Facilities * 

v .8. Federdl Bureau of the Census / K * 

9. Social Security Administration 

* 10. Pennsylvania Department of Commerce 

11. Pennsylvania Departments of ilealth •* ' 

12. Pennsylvania 'Drug and Alcohol Council , 

13. University of Michigan*' & RRRI \ \ , 

Ifr. University of Florida \RRI v < 

15- Oklahoma State Clearinghouse forf vocational .rehabilitation materials 

- 16. I.C-A, ^Clearinghouse ^Ln New York for vocational rehabilitation material 

17. Utilyersit>y contacts^/ at University of Pittsburgh, Penn" State, Temple, 
and Penn State'^Capitol, Campus f 4 

'*• ' A ' * ' ■ . ■ WHAT' ' 

6. To justify ex&enditures in terms of program-or project accomplishment . 
t, V • - - ' 

: - • x • • • how . 



*5 , 



a. Measiire % anei prepare evaluation data on. client outcome from the follow-up 
surveys,. 

b. Developed a methad.to prepare reports for the Client Assistance Project for 
the Grants and Facilities Section. ^ 

. ' c. . Respond tp external and internal requests for statistical data. 
^ d. Prepare and disseminate Profile Analysis-on a state and district * of fjlce 

level to increase program effectiveness, / ' 
► • e. Prepare numerous activity reports fior RSA and SSA\ 

* Prepare* and measure tjhe process and outcomes of the case review praeess. 

^g. Prepare and measure^ program °experiditures oh a. cost benefit basis as per 
• dollars saved in reducing or eliminating income tax recipients and ^making 
* 4 them taxpayers. ° \ ; „ % x •* \\ 

. - r < - I 

1, r 7* To-forecast fyfure resource needs ^.servi^e demands, for short ^ai^i* long range 
prbgram objectives. ' m * \* . ^ ,v> .. . * 

|; , ■' ' , : ■ ■ / ' •/ " : \ 



■ HOW ■ . " 

% * » * 

a. Program Evaluation Technical Assistance Center - review journals and other 
documents regularly, " ' 4 

b. Develop new reports, e.g. severely disabled report. • , 

c. Prepare numerous activity reports for RSA. '* / 

1, RSA-13 *- Vocational Rehabilitation^ ^Program Progress Report „ 
-2, 4 RSA^lOl - Quarterly Cumulative Caseload Report * ^ 

3. R$A-30_- Annual* Repx)?t on State Vocational Rehabilitation Agency Reviews 
4* RSA-102 - Quarterly Report of Clients in Special Target Groupa 

5. RSA-32 - Annual Report on Post-Erip'loyment Services^ -v 

6. Fe'deral Reporting' of the Prpgram Evaluation Standards 

7. RSA-1^ V Program .Financial Plane * * 

d. ' Prepare 'reports for the Governor's Office^of Budget and Administration 

1: Impact Indicators Report ^ 

2. -/ Projection Report f 



1 



■ V 
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- FISCAL ACCOUNTABILITY . ; \ 

' * • ■ v 

# Development- of a budget fop government grants <$r cpntracts' is 
usually the fir$t pha'se or consideration when embarking on § project, 
( ' * This means that prior to submitting the-bid, considerable detailed 
, thought and planning is k necessary to initially arrive "at figures, that 
represent a budget. In short, it must be determined in advance what* 
theldevdlopment of a contract budget may total even^before the ^i>id/ican 



be presented. " * f 

o • * ^ . 

*As for monitoring a project budget, it can be sabi ^^im ply that this 

employs the knowledge an3 experience of What hasf t^ken platce up { to a ' 

certain point in a contract; evaluating current results, and then deciding • 
* %• v f ^ * 

whether to continue with existing plans and procedures, or to alter any 

activities that may attain more -desirable results, ~ 

* There are numerous* ways this caft be achieved. Important consider- 

ations are, Vhat data and information can be-thevmost useful? Travel 

expense, ^airfare , lodging, subsistence? Who, and what groups, are bene- 

fitting the program most? Is preparation time too lengthy? IS reporting 

time and preparation, draining time that could be used to move the contract 

along at a better pace? _ 

* » 

From - these, and^similar questions, a determination .can be made ori 
how data should be gathered ^artd reported. It is frprn these factors -that # 
the forms and reports should be .designed and implemented.' A regulation t 
standard form for this type of reporting is sel'dbm recommended' due to the 

fact that moist federal lor $tate contracts are unique in purpose, and 

■ ' * I * . - , . ^ 

characteristics found -an one*,vdo not necessarily coincide with those in. 

* ■* » 

o-Jrher contracts, \ 

'It is well to "note that close scrutiny and up-to-date repo^^ig. will be 
a.,ma3or factor in attaining the greatest benefit f.^om a standpoint of. 



' monitoring a contract budget. As an additional aid, it is advantageous 
to meet with other staff members and discuss vario,^| Activities in 
connection with a project, . - 

'while rtnich more could be said, and,^ in greater detail on the above, 

i¥ these- concepts are generally followed, a satisfactory 'budget for many 

" $ . 

projects can be, developed and monitored. . k . 

Following are. explanations of several forms utilized* with appropriate 
charts: * ' • 

\- A- "Da£a gathering )in the preseat MEU contract, which the 

s * Commonwealth of Pennsylvania began with -the Individual Tim$ 

* * « * » — 

Report, by task used hy^«^ch staff member monthly CSee Chart 1). 

. B. Accumulating this/yirfa monthly -enables-the. administrator to 

maintain control currently at any .stage of the contract (See 

* . ■ * 

Chart .2 ) . • - 



Cc Reconcilement of personnel labor costs^, s*taff travel expenses, 
• cpntractural (consultant) expenses, otherx incidental expenses 
incurred; tan be achieved by the use of monthly breakdown 
chart (See Ct\art 3). 
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. • „ • MEU Technical Assistance - Summary j . "* 
by ; West 'Virginia "Research and Training Center 



MEU Technical Assistance Summary ^l/' . 

Pennsylvania Bureau of Vocationa-l -Rehabilitation 
I . December *£-6>-1979 (1', day T. A.) . ; 

( * Task 1 * » < ' - : . 

Present: (For part <^r all of Meeting) . * 

^Pennsylvania BVR' - Hagan, .Guise,, Jenkins, Chopak, 
, v. 1 . Hossler, Gfrib and Li?ng1 e" 



• * . WVRTO Nida . • . » v 
Agenda: 'Reviewed "program to date in eachf task: t . 

- I • Outlined: technical assistance needs., j 1 

* , " * '\ . / # < 
m < Prioritize^! task 6' projects according* to ^fol .Idling 

, % crit3na: 1 

- A. National significance \ ; 

# B*. Utility to agency , 
/" C. Probability ctf completing within contract / 

■year*...' * - \ ^ 

* * , • . * ... > 

\ Action Items: WVRfC - arrange for technical assistance o&lhe 

* use of general ize?. computer statistical 
/• 'package? and custom, statistical analysis 

• / .'of specific data bases. / 



tr. 

3J&*V f - 
J. 



• ft 



MEU Technical Assistance Report Summary #4 

S1x*MEUs- : - 

January 8, 1980. '(a day each) 
Task 7 

• ' • 
.Present ■ Delaware - Guy Veach 

<• * Michigan ---Bob Struthers, Ph.D. 
• * Mississippi .- Wendell Cox 

. Elizabeth Kirkland 
Oregon - Neil Sherwood 
Pennsylvania- BilV Jenkins, Ed. D. 
Harry' Guise 
.' Virginia '- Bill Brownifield 

WVRTC - Richard A. Nida, Ph.D. ' 
. • RSA - James E. Taylor, Ph.D. 



Agenda:' Discussed various dissemination and* utilization (DU) models 
and, settl ed oh the Sherman .& Weaver model . ' 

! < ' - 'A ' * ' 
• Discussed the requirements of "the del iverables- for DU and 
agreed to submit a joint plan for Task 7.2 and individual MEU 
' . plans for internal DU. . , - ' • 

Specified t^ie objectives of $M external DU plan,. 
Made a tentative list of MEU related, products . . 

. : : * . « Identified and categorized the- audience^ of the information. 

produced. • 
' «-. . 'r'.. . 

Listed the . Hnds of vehicles* that might be used to dis- •* 
. .'.«mi natfi "the information. ■• \ \ 



iff. 

w 



la:'- 




Listed; the effects that the MEUs. wanted' to .produce in 'the 



individual, audiences. 



■ • Developed a schedule* for the writing, editing, .and review • . 
of the External- DU Plan.- .WVRTC to c6mpile. MEU to review,' 4 
•WVRlC to print and submit %o RSA by March 31,. 1980. 



Action Items: RSA-,. Request change in deliverable date from' 
January..3l, 1980 tofMarch 31, 1980. . 

fpf|4-#.rv..'- . ... WVRTC -'Prepare draft External DU Plan for MEU 

i^and qorpent. ; 



-A" tons Wer^jtion^ai 




A 



J. * 



Present: Pennsylvania 

•West Virginia University 
WVRTfc ' 



**/'••*. Pennsylvania MEU 

'April 15, 16/1980 (fwp Days) ^ 
' \ [\\ Task 6- 

*■ " Will 1am Jenkins, Harry Guise, 
v • B1l> Shildt, Art Lingle, ' 
Dan Grib, Dan Hossler, 
Leah Kuhns, Harold Seitzer, 
, Zeldef Peters and Tammy Parti emore 

Oeff "Messing % '• 

' ' ' Richard A. Nida 



. ^Agenda: ' 




1< Reviewed each staff members vita, job descriptions 
and training history. 

2. Reviewed task assignments in group, setting. 

3. . Interviewed each staff persdn concerning 

task assignments and perceived training need$. , 



if? ■ ■ ' f 



Action Itefljs: 



p 1 



1. - jJePff Mes-sing*, in conjunctior 
prepare a training. plan for 
. with suggested training nee 
•v for meeting those nee"ds. 



with WVRTC, will- 
the MEU complete 
and alternatives 
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1;;; 




*• 
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* 


Present 


'» *- • 


WVRTC 


lit - 


' « 


f 

"„ » 


Agenda: 


'* 4 
* 


\ 




Action 







MEU Technical' AspS,|ance Summary #15 .. * . s • 

* Penns)|ia ; |ia MEU , • ' .* ' 
April «8,&J) (One day) , • - /" ' 

■M4- 6 . ' % "\. . v 

II •• . * Bill Jenkins, Harry Guise., 



Pennsylvania 



Andy ChopalT, Art Lingle, 
Dan Grib and Dan Hossler 



/"* Don McLaughlin 



1. N Reviewed Pennsylvania MEO plans for Task 4,5 and 6 
data analyses. 



''5. Presented possible alternative approaches for conducting 
data analysis. 



1. Pennsylvania Will prepare specific requests for 
, assistance »tib meet their research design and 

statistical ana^sfs needs,: , 

2. • WVRTd|igrefed to'respitod to expressed needs where 

appropriate. ' , • . 




A- 




' VY 



MEU TECHNICAL ASSISTANCE SUMMARY' #17 

♦ . Tbe_Si.x. MEUs * 

May 13, 1980 (One day to each MEU)~ 



fisk 4 



Present: Delaware 
Michigan • 
.Mississippi 
> Oretfon' ' . 
Pennsylvania 

<■ » 

, . Virginia - 
1 

.' WVRTC 



V 



Independant Consultant 



Guy Veach and Earl Tuberson 
Bob Struthers 

Wendell Cox and Beth Kirkland 
Lau'rilee Hatcher, J 

Bill Jenkins, Harry Guise, 
Bill .Shildt,. Andy Chopak v 

Bill Brownfield, Ben Kauffman," 
Jack Hayek, Janet Slipow : 

Rick Nida, David Whipp, 
Don McLaughlin 

Linda Barrett 



Agenda: 

• •' k - 1/ 

• 3.' 
Action Items: 



Update on progress of Pretest. , <. • 

Discussed possibll data analysis. . » '* 

Discussed* deliverable format for pretest completion, 

' .'V , • ' • ' ' • * 



; BPA fecr prepare map of standards data elements- to -elements 
: irf pretest forms. ' 1 / ' 




•6J5. 




* r - ^ 



i * 



■4* 



A. - 



MEU /TECHNICAL ASSISTANCE SUMMARY #20 

'-.».'. Delaware MEU • 
'• July 15 ^ 16, 1980 (Four Days) 

Task 7. -' 



Present: Delaware! • 



Washington, DC 
v r Montana >- 
Marine 
Nevada 

*_ 

New Hampshire. 



Ben Barker, Guy Veach, 
Earl Tuberson, Martha Jackson 
Vincent Finelli, Lois Jackson, 
Cliff Crouch, Bob Snider and 
Gabriel Markisohn 

Vernon Hawkins* ar^ David Songer> 

Joe Angel 1 ; 

Richard Ault 

John Griffin * 

Patricia Trow 



Rhode "Island 



Joseph Farrell 



IS? V'?/ ' 
Aaenda: 

§|P Action 

te> ■ . . 

Ife. •'■ • 



\ 



i 



Alfred. Kepnes-Lepourneau 
Robert Lindsey 
Tony Manuel 

Harry Guise and BilT Jenkins 
Richard Nida/ Nan Brerizel and .David Whipp 



'Vermont *\ 
'West Virginia 
Wyoming 
Pennsylvania 

I • 

wvRirc. 

See Attached ' . , . " " . 

VWRTG'inade presentations concerning" the role of the coordinating 
contractors, the dissemination amTutllizatton* of information, and 
building new evaluation capacity-. • Additionally- Dave Whifip took 
photographs, and prepared a story for the "MEU newsletter. • - 



Item's: 



Individual states - share W'conference information with other sjaff. 
membersi in .their organizations. * • - • 

Delaware MEU" - Plan similar, conference for FY. 83. ■ >y 
yVRf C .- Prepare slides "and pri'nts of 4he conference. , 



.V 



i 



.: 



MEU Techinical Assistance £Aimmary///2-6 ! 
' . Pennsylvania MEU 

September 7 17-18, 1980 " • ' : 

^days LAl) 
' Task 6 • 1 



Pre^sentj 



Agenda;. 



•Not applicable*.,- This technical" assistance Vas 
provi'ded by Don tafcaughlin from the WVRRTC in 
Mtft&frntowii', 'WV, without visiting the Pennsylvania 
w MEU in..-person. , ; t- 



fcjti - - 



This technffial^assistance consisted of technical* 
consultation^ and computer based 'analys es - of t Pennsylvania 
client data. This" ass is tance was directed at a detailed • 
examination of cases closured as Hjomemakers in 
Pennsylvania in an. effort to ^xplore and, to the ^extent 
possible, to isolate the ca*uses*qf the agencies high" rate 
of this type of clos&re^in the past. Specifically the 
analys-es conduc/ted as part of this activity focused ; *on 
direct comparisons of the agency 1 s' Homem'akers closures t 
and competitive employed closOTS^s in adcPiticpn to norf? — ^ 
successful clopmei cases (28s) ^ £s a result of 'thifs 
technical assistance the agenc^ w'aa pr^^ded vith {' ( 
several^ sets of analyses which should- set soitfe light 
on° the .Horaemaker closure situation in Pennsylvania 



Action Items: . IJonei 



\ 



.68 



'#* . v ' 
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Coifcpreh 



HEIf Contract ^Number HEW lQSf-7 8-400$ 



A* 



ehensiye Stat'e VR program arid policy System Tjir5ugh ; f 1' 
•A Mode\ Evaluation/Management Supp&rt Urti£ ' • " " -V*/ i , 



TASK IV 'DELIVERABLE 



PA CRITICAL REVIEW OF ^HE; ' 
( BPA PROGRAM STANDARDS REPORT 



V 



26 June, 1981 



^Project Staffs ' /; 
' Clark D> Kennedy, Project Director 
' Harry «W> Guise,' Project Manager [ > 
/Andrew W. Chopak * : . *; 
feV " Arthur C. Lingle - ' ,{-. 
.Donald E.^ flossier 



•Prepared for: \-\ ' 

Rehatilitafioft Service's Admin^ptt^ation . " f 
Office^ Human Dev^Qpment Sefvi^^s 
Dep^ttment of 'Educatllqh z 



HaroWM. Seltzer 



§k Alfreds; Newian 

Gl Wliliam Schildftl; 
T.amara.M. Parth'embre 
Zelda M> Peters 
Susan, H* Bechtold* " 



/ 



Pa. Prog^i Evalu*^tlpn Section, ; « 
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' * - - PENNSYLVANIA MEU REVIEW ON THE BPA» •> 

J: . REPORT- ON THE -PRETEST OF THE REVISED VOCATIONAL REHABILITATION ' 

f%, - /'».'.* - ' • PROGRAM STANDARD '. / j. % O ? 
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PRELIMINARY ftEPORT 
'VOLUME II (DRAFT) 



• - The conceptual' design of* thp Program Evaluation Standard^ requires "a system 
, ±n, " h&h . the "ates collect and. report data to RSA which, in [ urn , analy2es 

|, . retums;the results'to the states. To this overall systL, w | have -a few questions: 
fc; ^ Will, there 'ever again be written regulations for Program Valuation Standards " 

4.IV the Rehabilitation Ast of 1973? " * - S 

2. Should programs standards be set even in the 'absence? of means-of, measuring " 
accurately and. even when suitable sanctions against offenders do no't exist? 7 " ' 

3. How uniform shoulc} program 'standards be? 

A. How 'specified should program standards be 
I ' »* - , 

■ . — * U ™W» to each of these questions in the numerical 'sequence in 'which 
they- were presented. . %1 " * , 

. t £ - 

O * * - " \' 

^ . We feel that as-long as the PrVgram>aluation -Standards are" not a part of the 
!| : - law (Rehabilitation Act), they will nev^r .have the power. to change policy within 

•We realize that each of, the eight performance standards has' data elements that' 



a? 



is; *• 



if 



ffc". " y ^vquHtlf^ably measure^ach standard. The proble&as we "see them are: > are' 

m. ... 



we choosing' the right debasements tod' should there be a performance range? ' If 
'there- is a performance range, then. what type of suitable sections should be given 
to states £hat perform weil, and vice versa?* : " * . 

^ 'We thfnk. that program standards should be uniform but flexible to vary from', 
state, testate. We agree with BPA's process for setting performance expectations " 
.a^d • theif suppprt^ve evaluatio^ystem.. The supportive evaluation "system would be 
^Kykp to determine, why "a state agency\*as a low leVei of .performance, and to make 
^yecoWendations to imptov^ that ' agency 's performance. The bottom lifte is that ' - * 



Is?" - 



V every, « at , agenc, shoul, collect eht daca in . unlfora Bannerj ^ 
should be' analyzed in different ways 

There are times when- there is too little specification of programVandards 

"and times when there is too mu.h. It is difficult to make a blanket, statement as 
to whether, at thi, time, we need -more or.less explicit program -standards. « is 

- Perhaps ^too easy to suppose that' program standards will help where there is a 
needier -them, and that W will be neglected' where there is no need. If you 
wanTp-rogram standards evaluation to* be useful and fair, then you must have a sen- 
sitiyity to your audiences.' * * - * ' V 

_^ ^ We^fll react t6 BPA's Recommendations using the following keywords:' * 
(1) Agr^, (*') Neither Agree- nor Disagre'e, and (3) Disagree with Reasons Why. 
(Please refer to the chart on^the following' pages.) 











B?A f S RECOMMENDATIONS *OR PROGRAM PERFORMANCE - STANDARDS DATA ELEMENfS «• 

* • • • • x \ . 

' • / "* ' 


♦ Data Elements - 

' s~ : 


BPA Recommendations 




PA Reactions 


Standard' 1: Coverage 








(l) Comparison of caseload' served 
t to expenditures 

(ii) Clients/served per 100,000 
* population 


Eliminate ^from Standard. 
Retain as information item 
+ v . *fqr RSA.y 

Retain as is;. • 




♦ 

. , AGREE 

^ AGREE ^ 
< • 


(}ii) ^ Percent severely disabled 


New data, element'. 




* 

\ AGRfiE ^ 


Standard 2: Cost-Effectiveness and Benefit-Cost; 








(i) Expenditure per competitively 

employed closure . „ > 

■ . • * * ° - • * • 


: * 

Eliminate .from Standard. 
Retain as information item 
. for RSA. f , s \ 




AGREE . 


(ii) 'Expenditure .per 26 closure * > 


.«» 

Retain as is. 




DISAGREE 


(iii) Benefit— m<:t ratih 

• 


k 

* ** 

Retain as is. 


NEITHER AGREE 
NOR DISAGREg*- 


(iv) Discounted net present value 
Standard 3: Rehabilitation Rate * 


* Retain as is. 




v ' NEITHER- AGREE 
NOR DISAGREE. 1 

/ '. 


i(i)^?>J?ercent 26 "closures * *\ 

(ii)^ Arfftual change in 'mimber of ' ^ \ < 
l x 26 closures 


Rfetain as i-s. * # ^ 

V 

• New data element*. < 

0 v • 




AGREE 

NEITHER AGREE' 
U0R. DISAGREE-' 


Standard 4: • Economic" s *Independence * It 


• 






(i). Percent $6 closures^ with weekly 

earnings at /above federal^ 
) minimum wage 

k ERIC v. 4 v _ ^ \^ . - 


t 

Retain as is. . - ' < 

y , ' ' ' -Slfe " """ 


* 


DISAGREE ' 

> t V* 



pr/y Kecommenoations 



%A Reactions 



(ii) * Comparison of earnings of s 

> competitively employed* 

. 26 closures to earnings of * 

\ - . employees in state 

'(ill)* Comparison of earnings before *C \ 
and sfter VR services 0 » * 

(iv) Comparison &f 26 closures with 

public assistance as primary source 
of ^upport, beforer and after VR services 

tStandard 5: a Competitive Employment 

^(±5 . Percent 26. closures competitively' 
■;if » employed 

(ii) Per£ent>' a 26 'closures with hotirly * 

earnings at /above federal minSnym wage 



(iii) Percent 26 closjjres^ with competitive, 
- outcome 1 or with non-co"$ipetitit^^" 

, • outcome and non-competitive goal 

• v * 

(iv) Percent non-cpmpetitivfcly employed 
26 closure s 4 showing impjrbvement in 
gainful activity b 

>• • . ^ . , i 

'Standard 6: Causality 



Retain as is* 
EXijminate 



Eliminate 



♦ Regain as is» At ^ H 
Eliminate > 



AGREE k - 
AGREE . 

AGfcEE s 1 

ACJREE * 
DISAGREE 



'Move tq*Pxoc$dural Standards. . AGREE * ^ 



Modify definition of 
"non-vocational improvement" 



AGREE \ 



\ 



^ Comparison of earriings^change from 

Mk€- refe ? ral ;to closure of ,26 closures 

* to earnings change of a v control group 



Standard ^iiminatedJ' 



AGREE- 



a • 



fjf*t>ata Elements 



?A RQcbmmefldatrions 



PA Reactions; 



IStandard 7: Retention of Benefit^ 



•(i) ? ^ Percent of 26 closures retaining 
lw> Earnings at fo;llow-up 

Comparison of 26 closures with 
public assistance as primary source 
of support at closure and at follow-up 

|(iii) Percent non-competitively employed ' 
g|>' * #26 closures retaining closure 
* * skills at follow-up . 

^Stah'dard 8f Client Satisfaction ajid Utility Appraisal 



Retain as/ is. 



Modify definition of 
."primary sourcte of support" 



Modify definition of 
• "non-vocational improvement" 



-Percent closed^ clients Satisfied with 
|fiv>, overall WR experience / 

iCil) \ percent -'"closed clients satisfied , 
lfe^^ w i* ft specific aspects of .VR . 



Retain as is. 



\ 



- Eliminate '"satisfied with 
present situation" 

- Combine counselor satis- 
faction items. 

- Retain service satisfaction 
items 



Combine wifc^ 8(iv), 



|(i^i) : Percent 26 closures judging services 

*. received to have been useful iii 
fSi>" 1 obtaining their jofc/homemaker /situation 

^ Pefce^/26 closures Judging services 
mlB .received to^ be useful in current 
Ip*^ ' performance £n the jott/homemkker , 
fsflf . : ; .situation W'-' • . 

life . . i - ■ 

» ' : - 

JSSBpr. » • . . . " • ' 



Combine with 8 (iii). 
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AGREE 



AGREE. 



AGREE 



AGREE 



AGREE 



AGREE 



AGREE 



ST*- - ' 



.Why we disagree with: 



• 1 



Standard 2: Cost-Effectiveness and Beivef itrCost 
(ii) Expenditure per 26 closure 

. We totally- disagree with your statement on page 36, first paragraph. Calcu- 
lating^ average cost of 26 closures is not at all difficult, since the All 
Service Cost Total item' is included on every client served in the agency and this ' 
|; ' information is sent to RSA via the R-300 magnetic tapes oft. an annual basis. The 

statement "The extent to which agencies incur expenses, with undesirable' results is" 
a measure of the inefficiency of that agency" is not true. Just because a previous 
client dies, or moves out, of the st^te should not reflect a measure of inefficiency 
for that agency. The RSA-2 report gives expenditures for just the current fiscal ' 
year, since 'the number of 26 closures during the fiscal year may or may not have 
cost incurred during the same fiscal year and the year of closure, we feel t ha data 



element is mea^Lngless, If you -seally want to address this data element, you should 
divide the All Service Cost .Total item for all status 26 closures' from the R-3.d$sT 



f|^V_ magnetic tape by the number of 26 closures, 
Why we disagree with: 



• Standard 4;" Economic Independence 
(1) Percent 26 closures with weekly earnings^#/above federal minimum wage 
^We^disagre^ with your statement oh page 55, next to last paragraph. Any state 
agency th^'-hai a large numbefSof homemakers, "sheltered workshops, or part-time 
Kr. ^^ 1 ^ ^ 16 ^ <20 hours or less) will always show up . poorly for this data' 



Mr ,. 



.,: element; ' Since the 'Rehabilitation Act. defines 26 closure as-suitable employment, 



th>t you are treating homemakers,; etc.,' as'if they were non-productive * 
iostire^J :Here is a .better idea: . Instead of dividing by the total nuniber of * . 
K^^y? t ^ 6 ^ aOSUreS, • fi il C -? ubtracC ^1 homemakers,: sheltered employment, and part- ; 



^^^ fi ^^^Wlo W t from the total 'status 26 closures, then divide; .' Now, 

management tool; ' , 
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Why we disagree with: 
Standard 5*: '* Competitive Employment * ^ 
^ (ii) Percent. 26^closures withjiourly earnings at/above federal minimum wage ' 

Instead of eliminating this data element from Prdfcram Evaluation Standards; 
i we would* like to see the data element modified. " ? 



*5 «• 



?: 5* o 



• Number of status 26 closures competitiVely 

n - ' employed (work status at plosure codel, 3, 

'Percent' of. status 26 closures . and 4) with hourly earnings at/above mini- 

Competitive employed with \= _ mum wage 1 

''hourly earnings at /above Number of .status 26 closures competitively 

< federal minimum wage . employed (work status at closure code 1, 3, 

> . • . or 4) * 

• , » 
We disagree with statements made on the following" pages:' • v 

Page 20, first paragraph '- the words' "ratio of, severely disabled" should be 

changed to read "rate ^severely disabled"} Also, 'in the . same paragraph,' the' ! 

example "Status 10 and above". should be changed to read "Status 06 and above". 

- Page 21, last paragraph - "Status 10'and above" should be changed to' "Status 



jp". 

|1' 06 and above". 

P" ■ * 

z 

ttr 



* Pa S e 23 > second paragraph - Beginning with the words !t We are aware... 11 , we ' 
disagree with the first sentence in the paragraph and the^last sentence in this . 
■ paragraph. No' state agency knows a client's case is going to be closed 08 when 

/* . * * . 

* tSe. client -gs receiving diagnostic services/ A knowledge of how the referral 
: process tjjg 'the VR" agency works would eliminate a* statement, of this nature] 

Ir * ■ • *•." ; f* 

. Page 26, third to last paragraph - We in Pennsylvania recognize client assess- 
|| ment as client service Any case-'thaft uses VR dollars and/or Counselor tiWe should 

f& ■ • ' • * - ; • " / ; - 

^be galled a served case : . g „ " . v • ■ < 

Page 46, first paragraph - "The second point concerning Standard A" should be 




Page 67,*. last sentence on tffe page - We in Peifnsylvania realize that u goal- 
switching" is not a bad thing. The Pennsylvania case review process has shown r 
'that our counselors have justifiably documented the reasons why they changed the 
vocational objectives, 4 Our counselors have amended the clients 1 IWRP's appro- 
priately and /have provided the necessary services under the amended -plan. ' 

Page 70, last paragraph - /'Dividing competitive employment into full-time, and 
part-time employment is complicated and unnecessary. *^Et adds virtually no new 
information, and the variation, in hours worked is already taken into consideration 
ill other data elements. 11 The. first sentence is not true. Distinguishing full-time 
and part-time competitive employment is important to our Q agenc?y, ,and if i's not 
difficult to calculate. It does add new information, since BPA is suggesting to 
eliminate 'Standard 4 data element (ii). 

• * ■» * 

* PENNSYLVANIA MEU REVIEW OF BFA DATA-* COLLECTION FORMS 

I. We disagree with your.recommendations to retain Item W: Address of 
Friend or Relative on R-300 Supplements A & B. In the pretest, 52 out 'of 100 

***** cases had this item completed. Many counselors informed us that many of the 

• i ' • < 

clients didn f t want to give an address of a friend or relative because the^ did 

t " . . . " ■ . , 

not want ^nyone to know they were receiving services fro^our agency. We feel 

• • * ■ » 

that this item should not be cofl^uterized. because it creates a burden for the 

t. ■ ■ , A - . ' ' - 



5^5, 7 



counselor_to collect. We see no utility for our agency to Qollect this piece of. 
data on our clients. 1 



II. We disagree with your recommendations' to retain Item Q: Primary Source 
^6f Support at Closure" on R-300 Supplements C & D.\ In the pretest, 231 out of 503. 



cases indicated that current earnings, interest, dividends, and rent as their 



.' ' primary source of support at closure. The only major difference in primary source 
of support code seemed to be between, the 26 closures and .28 and 30. closures. The 
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primary source of support code answered most frequently for 26 closures was • • 
current; earnings whereas for 28 and 30 closures was some'fc^m of public assis-- 
.tance or, SSDI. In addition, this item was very difficult "to capture in the 
case file, since there was nonstandard place in the case file to look for this 
information'. Many tira« the only reference to the primary source of support at 
closure Was .the same as the primary source at referral, especially for not reha- 
bilitated cases and homemakers. For rehabilitated cases, 214 out of 300 had 

current earnings as" primary source of support. The remainder were homemakers 

— * * ft > 

which had family^and friends as their • primary source of support. It is very 
difficult, for a counselor to get this information from a client whose case lias" - 
tbeen closed not rehabilitated or rehabilitated as a homemaker. The problem; as we 
ke i£, is with the words "Primary ^Source of Support". We think- it would be more 
meaningful- if all state agencies collected a-11 public sources of support and dollar 

■ 1/ ' - \\- : '. 1 ' : 

amounts at referral and at tirte. of closure, therefore an agency could measure the 
degree of dependency of VR clients on^monies from public sources^' 

III.. He disagree with your recommendation to delete Item RzCrhunber of^Hours 
Working Per Week on R-300 Supplements C & D. In the prete'st, this was a fairly 00 
easy item to collect since it could be located -on the Closure Amendment of the 
IWRP in'.the case file; This- item will be' an. asset to program evaluation since it 

will distinguish part-time competitive employment from full-time .competitive 

' • ' •• '.':*. . " • ) 

employmeat. We strongly emphasize that inf ormation' 1 on this item be collected on - 
status .2-6 closures only. Hours, worked per week is -difficult to capture on riot ■ 
rehabilitated cases and even . if this information is captured, it is usually zero 
hours j ' . , • ' ... ... . 

. .1 * V; ; We a § ree w * th y° ur . recommendations. for both the' Closure Survey and . • 
■ Follow-Up Survey. We would like you- to consider ? our new "Follow-Up Survey" ^that, 
was- developed based upon the* field testing of the Closure and Foliow-Up Surveys. . 
.(^•e -Attachment A) We have data available on the reliability "'of this new , - 



instrument,. (See Attachments and C) 



„ PENNSYLVANIA MEU COMMENTS -ON INTRODUCTION CHAPTER 

¥ * ' ' • \ ' V 

Generally, the introduction chapter was very clear!, and briefs There was a 

* " \ * « 

* * spelling error on pa^e 5 under the title Reporting System , The word measure is 

'* "* » 1 * ♦ s . , * 

1., . ' spelled incorrectly. We feel that one* of the best^ sentences in the introduction 
| . chapter is on page 12. - "The standards have* been dsed for reports, but not "as an 

incentive to improve performance." If the new program standards can act is an ♦ 
agent to improve agency performance, this field test activity would have b§en 
worth the effort/" m 

PENNSYLVANIA MEU COMMENTS ON. GENERAL FORM AND PRESENTATION ; 

• / . 

Overall, the general Ijrorm and presentation was excellent. There are two 

y \ 

areas that we feel should have been addressed *but*were- not ♦ A definition of com- 
petitive employment was never stated. Our question is: Does competitive employ- 



ment include RSA work status code 1, 3,«&and 4 or just, code 1? The other point that 

w" * . ■ r ~ " • / \ 

* we were concerned with was Standard *2 data elements (iii) knd (iv) ♦ We would have. 

!£- * * " ' / . 

C" : liked to see the formulas Used and the actual data used* to \compute the statistics 

♦ . - * * * _ v 

*shown on page 43 for data elements (iii)- and (iv) ♦ { _ w 

RS- •-•»•• . • • i 

iwr Since, for the most part, our state (Pennsylvania) has' stated the things that 

W : . . ■ ■ ' - ' ' ' '. 

we disagree with, there were faany good points that the report addressed.. In fact, 

the report was very well writsten and Pennsylvania MEU gives' it .a grade of A-. 

[This is based on a scale of A - F,. with A b$ing the best. . 




COMMONWEALTH Of PCNNSYlVANtA . 
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DEPARTMENT OF LABOR ANO INOUSTRY 



ATTACHMENT A 



iUREAU OF VOCATIONAL REHABILITATION 

x . - HARRISSURG,-PA. 17120 " ' 




FOULpW-ijjP 'SURVEY 

Phone where you can be reached; 



New Address: 



1. 



Our records indicate that you' may have received services from the Bureau 6/ 
Vocational RehqbtiitQtion. In an e//ort to improve services we provide, we would like to > • 

know*your current situation). We would appreciate your answering the questions on N 
this questionnaire and returning it in the postage-paid envelope as soonas possible. Ail 
.in/onnation is con/identiai and is /or research use only. No employers will be contacted. 
This utudy is bein$ conducted to improve services that the Bureau 0/ Vocational 
Rehabilitation provides to citizens. Please help by completing this form. ' 

Which of the following statements Jiast describe* your present jitiktion? (Please check one box.) 
. □ a, I earn a wage or salary at a regular Job. ;„ / • * 

I earn-a wage or salary in a sheltered workshop (for example, a place like a workshop for the blind or 
Goodwill Industries), - 
I am self-employed. 

I earn! wage or salary in a state-managed Business Enterprise Program (that is. a state-run program 
intended specifically for disabled people). ? \ ■ ' " 
I am a homemaker (that is a person whose primary woyk is taking cara of the home). 
I work on:a family farm or in a family business withoqt pay. 
I am not working at present. I am: (Please check one) 

□ 1. A student. 

□ 2/ Retired v ^ , 

§3. A trainee. - a 
4. Other (Please explain) l - 



□ b. 

□ a 

□ a. 

□ V 



□ h. My situation is not listed above (Please explain)' 



— IF YOUCHECKto ITEMS E, F. G. or H IN QUESTION 1, PtSASl SKIP OUBSTIONS 2 and 3^ ^ 



'f 3. 



About ho* many hours a week do you work? (Please cheese ana* fill in the blank) 

0*a* I usually work about : : hours a week?*"""^ 

□ r b. I am not working. 0 • . 

\ a O c. idoatknow.. . • j 

\VhaWre your present eariilng* before taxes? (Please check •one'hhd fill ip the blank ) 
□ a. I earn! — per (t. hour / 2, week / 3, <month\/ 4. year]. (Pleas* 



Q b. I am working but I don't receive a wage or 
' d c; <H anxiot working. f . ^ 
;0 4 : Idon*tkhow^ * - 



9alary. 



(Please circle one.) 




■I 



if 



Wfcat b your largest source of supfrort?*>/ease check one.l 
□ a~ Currant oaminn fmm a * 



a d 

□ e. 



□ 
□ 



O h. 
□ L 



□ 
□ 



□ L 



Current earnings from a job. 
Earnings from interest/ dividends or rent payments. ' ' / 

Family and friends. * I 

(f :°M 8X T P,e ' ^ §alvation A «* GoodwUl industries), ! 
^welfare „ possible. plea^heck which one of the following type/of welfare is your largest soucce of 

□ 1/ Supplemental Security Income (SSI) for the Blind « 
Supplemental Security Income (SSI) for the Aged - 
Supplemental Security Income (SSI) for the Disabled 
Aid to* Families with Dependent Children (AFDC) ' ' 

General Assistance (GA). * 

Social Security Disability Insurance Benefits (SSDIL • * 

Veteran's Benefits,, » ' \ * _ 

Private Insurance^ v " j *< \ 

unemployment .Compensation, V 1 

Other (PieoseeicpJoin.) / 



□ 2. 

□ 3. 

□ 4. 

□ 5. 



8. 



ZZt^^tZST' ^ '™ "™ °< « Que.Uoa 4e ? 

CJ a. ' I received $ — . Jast month. ^ s ° 

□ b. Mono, „ » 

□ c. I don't remember. y - 



i ' C v. - 



How satisfied are you with your present situation? 
□ 1; Very Satisfied , • • . □ 2. Satisfied 



.□ 3. Dimmed, □ 4. Very Dissatisfied 

□ ^Satisfied ^ □ 2. Satisfied' □ 3. Dissatisfied ^ Very Dissatisfied- 

SJeK^ 

□ 1. Very Setisfied i . . D^SaUahad ^..Dissatisfied • * *□ 4. Very Dissatisfied - 



□ 2. Satisfied 



pu? 



□ S^Dissattsfied Q 4. Very Dissatisfied 



f. 



U2. Satisfied □ 3. Dissatisfied , p 4. Very Dissatisfied" 



4. VSry 



Dissatisfied 



u a. x«§ , □ u 1 don't remembor . foc.No ' - — 

I^h. ^ *W»irt you with these senrfcW (Please check one.) 
? -p.l. Very Satisfied.. . □ 2. Satisfied « . □ 3. dissatisfied , 



•/Did y^co^Mlorproytte fob counseUa, aisl.tanceT^e check one.) 
w V . a Ye * . □ b. I don't remember □ a No ' 



4f yes, how satisfied wore you with'ihfs holp? * ■ 



□ 1. Very Satisfied 



□ ,2. Satisfied 



0 3. Dissatisfied 



□ 4a Very Dissatisfied 



.□ 4. Very Dissatisfied 
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. . TO HE ANSWEK^DrBY HQMEMAKERS ONLY 'bu,.^.' H.. He. gotten 

- _ . * „ tfresame Improved worts 

a. Caring for children and family ° „. n „. ,.._,.« ' \ i, 

> Caringforpets . ' J \° [ 2 \? P> ° ■ 

c Cooking andfocingmeals • ffi 2 • "2°. m 

1 £■ Washing dishes...;.. ' ?! 5 (2) D % 

t Polngleundry-...;'.. .•••/••••.•»•• .(1} □- (2) □ t 3) / 

g. shopplngf 9 rfoodendg;oce;l;;:.^^^'::::::::::::;::;:•■• »gfg go- 

h. " Running errands )]{ ° 2) ° t 3 J ° . =■ 

t- Budgeting and paying bin. s g° 

J. TaJdngpartlnnelghborhoodactlvlfle.- ..... .' ! J n gg • SS . • . 

j-4. k. Other fspedftrll!^_^ * N , *' u « l*J u . P) □.. 

<*• * . ' — wo-- (2)o (3j ti ; ;" 



□ b. No 



18* Would 



you wcanuneud vocational rehabilitation service, to • friend? (Phas* check one.) 

□ a* Yea " □ b. No \ " 13 1 



^.WouMyou^.,^^ 

* ~ yea, whjch services below do you require? - 

' "a* Further training ' . ' 

□ b. Placement assistance is findin/me a Job ° 
D c. A' hearing aid » 1 

. . - r Del Glasses t • 4 

^ V P Anew artificial appHanc8(e.g. leg. ann..brace, wheelchair, etc.)* ' * 

Dental Work . r - ' - 1 



Dental Work . 

□ g. Someone to talk to about training me and finding me a lob 

□ h. OtherfsDecifv) _ «*™*,oo 



Other (specify} ■ 

2<h Who answered this questionnaire? (P/eose check one.J 
□ *. I answered tnis questionnaire by myself. 
D b. I answered this questionnaire with the. help of anotner person, 
u c Another person answered this questionnaire for 

ll^nank^oufor y our c ooperation in thla'tur^uT you have 



/ 



me. 



any comments you would lik*Jo add, please include them here: 



I. • 



». • v* • - • . 

V 

■ * • 

• ■ * «• ^ 

Did your counselor arrange for you to have TRAINING? (PJeose check one.] , s 

□ a. Yes □ b. I don't remember □ ,c. No - * • \ ' »• 

If yes.'how satisfied were you with the Idnd of training you received? (Pleose check oneji 

■O 'I. Very Sejjefied ^ O 2. Satisfied ' □ 3. Dissatisfied J □ 4. Very Dissatisfied 

- 10 - %z^*z% ^7 ^ rece,ved from : he r^r . pr08r,m w ^ Y0U TO 

* a « * Thfi sendees were of greet use. • '* , ~ 

■ tS b. The services were ofsome use. - 0 

Tb° 8erv ices were of no use at ell. • v. ' ' • * 

. ( Od. I received no services from the rehabilitetion program. ■ \ *■ 

situation (eyen if they were ofjlttle or no use in helping you get it)? (Pleas^ check one J ■ , ' ' 

a* ^Thojervices were of great use. • % a > * 

□ b. .Tha services were of some use. " *) . 3' a - *• " » . _ 

□ c. The services were of no use at all. a i 
x ,□ e. I received no services from the rehabilitation program. - * , ^ 

members toN^o* or^vpTk longer hours at a fob they already have. Is this so in your case? (Pieosefcheck one.) * 
. . □ a^Yes a b, I don't remember A Q c. No 

% . « I 

StSK^ ""J ^ f aWUtttti ° n >"*^W th * ^titudes change. How do you 

fel about yOURS^F now. compared to before you were In the rehabilitation program? (Phase check one.) s 
I feel the same about myself. ^ 

□ b. 1 feel better ab/)ut myself. , * 

□ c. I feel worse abou\ myself. ^ * 

□ b. I feel better about my future. ' * . , ( 

□ c. I feel worse about my future. * * • 

« • . *■ . * 

, - »••*•. 

15; HieyourABnjrYTODOa-nyofmefonowmgfimcUoniofSilfK^chanMd^c. • ^ 

Has stayed Hat Has gotten *. 
"•""■«. Biting • * • tkeeoma Unproved' worse 

d. Grooming fsho»- w hair cart, rndke-up. etc.) ; m n *t ,{ n ' «n* *" 

e. .Dressing and undressing n I'O 

t Movingmy body weigh, J g So ' 

# f . Moving.eroundUiehomo •. Y« o>- ftl □•. fa O 

-. h^-€Umbin« stairs .... ' Y.l ~ ^- l ) 

^m. Moving on.the street : . ...... 71 * • • 1 n n If C n 

-■ j'i^n^.,...;--;:::::::::::::: ^g. -gg.-gg. . .. 

- - - «S" r: - r SS SS §g ■ * - 
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S }} a a 

June v l, 1981 



^ berkeley . planning associates 



s&E&^jv :v ~ 1 - Bureau of Vocational Rehab 
V . Blbor^:and\ Industry; Bui 1 ding 
• * 'Seventh and Fors^ter Streets 
H&rri$btirg, PA 



Dear Mr. Chopak: 



aj::. 



Enclosed you will find a copy of the.seco^ volume &£ Berkeley Planning 
Associates* draft Pro grain Standards Pretest Report. We would appreciate 
^very'mucjj if your staff woul4 v review the report and .provide us with 
J^ffr qpmments:* We ate particularly interested in your reactions to our ^ 
? — ^^^P. dati>9I l5 /J e g**4^. which of tKe Standards and data elements'should^ ^ 
*e reff&ied in" (or added' to) the final revised Standards system, and 3pT 
iecpnpendations on the data collection.^nstruments." Also,' however /we 
c , wodl<£ appreciate comments on the sketch of the Analytic Paradigm, (presented 
l|€t^i^ w ^ i> the*' introduction) and on the generai format^and presentation. - * 

, '.Your <?pmments wiLi be incorporated into our final pretest report, scheduled 
for completion' on, July IS,. 1981^ " Given ,the\ p timihg of the next report, it 
is impoi^^t for us to receive your comments as v soon as possible. At $l\e * 
Ve wpiild need to^have your comments by the time .of the next MEU 
f meeting in* Scotisdale. I. wflL.be attending \hat meeting, and willmpick < 
jthejn up\at tb^t'tiiiiQ if you h^avj^npt been able ; to send them -to me, before J 
then. So as. t<> minimize niisinterpretation, I Wtmlcl request; thalr your 
'..^efemme^j rb^ in waiting.* " , ,<f v • \ / 

, >i , * * x c , " - , , • ' - ' ' * # w = • * * ^ ^ - V 

'^W tbp^gg fon^ard to receiving your comm^n^s ^n'Fs/eing you at the conference^ 



Has* 



Sincerely', ' '* .. < 




^1 



SUGGESTED OtJTLItfE 




MEU . Review of BPA Standard Pretest Report 



life, 1 ,-. ;", 



I. Overall Comments 

■ * / * 

II. Reaction to BPA Recommendations 
*- a« Standards . * \ 

Y j 

p JData^ Elements 
/c. forms 

III J Comments* on Introduction chapter 
IV. \ Comments on General Form and Presentation 
Other (?) 



••■'• -'»•• V;. : 







sm^ia^ x 



JV 




APPENDIX 12 
^7 * 



SIX-MONTH FOLLOW-UP QUESTIONNAIRE AND RESULTS 



'-VS-' 



1/' < 




" BUREAU OF VOCATIONAL REHABILITATION 

' Labor and Industry Bujldfng , 

< » ' Seventy and Forster Streets /' 

Harrisburg, Pennsylvania 17120 , ' 



4 .■ 



: ' Our records indicate that,.wu: were referred to and may 
have received service's ffcTnr this agency. In an effort to 
improve services we^pfoyfde, we would like to. know your 
current -situation. We >ould .appreciate it if you would ' 
ariswer/tHe iew qu|s.tions t ohij5is questionnaire and return 
it in the postage-paid^self^aadressed envelope as soon as 
.. -pos^ble,; All inform is for research 

use only. Nouelriplbyers Wiirfeefcon'tacied. This study is 
,r -beihgf cbnducted to try to imp fovfe'serViOes to handicapped 
;r people: Please help by .filling out this form. '*« 

' - '*■•*< ' yse ' ' ' 





- 



IS 



EL " 



E i * " 



2-5 Questionnaire No. 

. - * VOCATIONAL REHABILITATION SURVEY 

«* 6-MONTH FOLLOW-UP QUESTIONNAIRE 



6-7 
\ 



k T 



(1) Which of the following statements best describes your present 
. situation? . (Pleasd check only one box) 1 
• □ 1. I« earn a wage or salary at a regular job. ' ^ 

□ 2* I e&rri a wage or salary In a sheltered workshop (for 
example, a place .Hke a workshop for the blind or 
Goodwill Industries). - ~ 

I am self-employed. . . - 



* □ 



.-art, • 
V.Hi"** 1 



I earn a wage or salary in a state-managed Business 
Enterprise Program (that is, a state-run program 
intended specifically W disabled people). 
, □ 5, I am a homemaker (that is, -a person whose primary 
1 ' work is taking care of the home). 

• P 6. I work on a-family farm or in afamily business withoutpayTA 

* □ 7. I am not working at present. I am: A student 

" □ 8, I am not working at present, I am: Retired ( 
, C3\9. I aim not working at present. I am: A trainee 

□ 10. Other (Please explain) _\ i_ • 

□ situation is not Ji$tQc) above. (Please explain) * 



8 c (2) How satisfied are you ywlth your present ^situation? (Please 
.•.check one) ,\ %, " * . " 

•V' 1; ->Veiy satisfied. P 4, Very dissatisfied 

.2'; Somewhat satisfied □ 5. No opinion ; 

*\EJ' .3. /-Somewhat dissatisfied */ • 

^ * ■ » * * * *. 

9-10 (3) • About hpw ftiagy hours a week do you work? (Please chedk one 
and fill in *thb\boxes) : 



hours a week. 



W-13 




I; usuajly worfc ab*put^ 

I'doT^t^knbw.; ' * / . ' ^ 
Jvam^7ioi*worl<ing/ 0 V ^ 

(4) • What are your present earnings? (Please check. one andlill in 

r V - *Ej; '* f* earn $ ( i I" lil f a 1 ; perlvyeeR. 1 ■; =. ^ " 
•Eg-'", r ' kdoB't. know. ' ■ - , . - . " * 



□ 



Pim . working >|t)ut l 4 don't receive a wage or salary. 
^4 *arn -not working r ' . 





ff? 



How long have you been in your present job? (Please check one 
l and fillTn[ the boxes) 

□ % . I haye,had this job about I 'months. ' 

I don't remember- 
er I am not working, 

16-17 (6) If you usually work, have you been out of work at any time during 
the fast six (6) months? (Please check one and fill in the boxes) 

□ :?No , I have hot beervoflt- of work at.any time. 

□ , J dpn-t remember. ^- *^»» 

□ , ' Yes, I was out of work for l_J_iJ weeks. c 
' t □ i don't usually work. . 

, * 

18-19 (7) What Is your largest source of support? (Please check on.e) 

□ 1. Current earnings from- a job. 
Earnings from interest, dividends, or rent payments/ 
Family and friends: <. t 4 

Private relief agency (for example, the Salvation Army, 
Goodwill Industries 

Supplemental Security- Incorhe (SS|) for the Blind 
Supplemental Security income (SSI) for the Aged 
Supplemental Security Income (SSI) for the Disabled 
Aid to Families wfth. Dependent Children (AFDC) 
General Assistance 1 (GA) -:* 
Other public welfare (Please explain) 



□ ~2. 
*0 3. 

□ A. 



□ 



5. 
ff. 
7. 



□ 8. 
0 9. 

□ to. 




D ; 11. /Public institutions (for exarftple, ;a tax-supported 
hospital, nursing home, treatment center, etc.) 

□ 12. Workers' Compensation - , " ? • 

□ 13. Social Security Disability Insurance Benefits (SSD|) 

□ 14- Veteran's" Benefits • ^ • % 
□.15. Private /Insurance \\ 

□ ,16; Other; (Please explain) v 



, 



20-23 




(8) How much income, if any, did you (or your dependents)" receive 
last month from ail sources of public welfare? (Please, check 
bfte^nd'fill in thfr boxes) - '* '* L' 




□; f received '$ 
*0 ; None r 
^ I; don't remember.- 



JhMJ- lasnmorith. 

- ■» 




24-28 *(9) ■ Hbiw^mu.ch clid you earn duri^gjhe last six (6> months? fP/ease . » , 

;;. r ' r Cl; ^ I earhey;4b6ut $ li L [ during the|iasF . ' 

V, ^SmontKs;- ,V J * ^' ^ 

/ I djd;nbt ear f n anything daring the last 6 months. ' 
IdonY remember, 1 ^. 



I 



i 




^1 




;>, $ - r-sv r 

W&t >- ■ 

% i - 
||;V 

•« ~* 

tS 4 ^ ... 

we* • 

of"- 



sf ■ 



29 (1t>) When you finished rehabilitation services, was anyone in .your 
family able to work or Work longer hours at a job they already had 
because you were able to care for yourself? (Please check one) 

□ 
□ 



2.' No. 

I don't know! 
1. .pYes 



-HP 



30 



31 



A. It yes, is this still the case? (Please check one) 

• 0 1. Yes ' 

* ' □ 2. No 

- □ 9. I don't ICncrw . 

(11) How 'do you feel about yourself compared to 6 months ago? 

(Please„check one) ' _ 

1 feeJ tfte same about myself as I did* 6 months ago. 

□ 2. I feel better about myself than I diet 6 months fcgo, 

□ *3. I, feel worse . about myself than I dicp6 months ago. 

□ 9. I don't know. / . 



32 (12) ' How do you #ow feel about your future , compared to 6 months 
' -ago? (Please check one) 
□ , 1. fjee^the same about my future as I did 6 months ago. 
«-» □ 2. # I feel better about my future than I did 6 months ago. 
□ 3. I feel worse about future than I did 6 months ago. 
Er 9. J. don't know. ■ ^ 

(13) Has your ability to perform any of the folkjwing functions of self- 
care changed at all during the last six months? (Please check on 
thffrigh&i8e,wheiheryour ability to perform^ach item has stayed 
tfyesame, improved or gotten worse during the last six months) 



In the last 6^monihs| 



v 



33 

,34 ' 

351 ' : 

36£\; 



£zf Feeding \ 

b;i? Bathroom functions , 

c. -Taking^medicine *" , 

Qrp9rnir1g\(shaving, 
'fiai^care,Mafte/i 

p"ressiog}andi ; / 
{und^singf- 'S 

^qyiog^my -body*** \ 
Weight* ' \ 




Has stayed 


"Has 


Has^gotten 


Not 




the same * 


improved 


"worse 


sure* 




1- □ 


2. □ 


3. D 


4.0' 




. 1. D ; 


2. □' 


m 

3. □ 


4. □ . 




'1, □• v 


v l 2. □ 


3. □ 


•' 4. □ 














2.0: 


3. □ 


.4. □ ' 






- 2. □ 


3. □ 


4. □ 








' 3. □ 


4..D 




















f 










f 












> 






t ' 


i ' ■■ 












" 

:| 






* 




; J 
. '4 




< 


' 39 


9- 


Moving around 
the home 




40 
41 . 




Climbing stairs 
^Reading 




42 ' 




fWtitia& 




43 


k. 


Talking 


o 


" 44 

» * < 


I. 


Answering the 
phone * ' • . 


* 


45 


m. 


MoVing on the 
sjre^t 




46 


- . n. 


Usincjpublic 
transportation. 




47 


0, 


Driving a car. ^ 



1. □ 2. □ ' 3. □ 4. □ 



1. □ 
1. □ 
-1,D 
1. □ 
1. □ 

1. □ 



1. □ 



2. □ 

i □ 

2. □ 
*2.0 
2. □ 

2. □ 

2. a 

2. □ 



3. □ 4. □ 

3. □ .4. □ 

.3. □ 4. □ 

-3. O 4. □ 

3.' □ 4.-D 

3.' Ch 4. □ 

3. O s .4v fa 

3. □ ^4. P 



WW'S.:'?* 




(14K If ypu are primarily a homemaker, has your ability to do anylbf the*" 
skills and activities related tfr homemakln g changed durfpg 
past six months? (Please check on the right site whether your v< 
ability to do each of the Heps has stayed the same, improved Qr ' 
gotteh worse during the last six months.) * 



In^he last 6 " 
months my ability to: 



Has stayed 
the same 



M8 


a. 


Caring for children 
and family * . 




49 


. • b. 


Caring for pets 


' 1. □ 


50 


c, 


Cooking and fixing . 
meals, ' 


1.0 






Washing dishes* 


1.*Q 


52 


e. 

■ 4 


Cleahing, house 


1. 


P.* * 


Doirig laundry - 


1. □ 


54 . • 


V 8 ? 


Shopping for-food 
and groceries 


-1. □ 


55- . ; 


.. h. 


Riinnlng^errands 


: i. b ' 


56 




Budgeting, and 
: paying bills. . 


.1. cr> 


?7 : 




Taking part* in , * 
.neighborhood 
{activities' < 


1.' □. 



improved 


Has gotten Not 
worse ^sure 


2. g\ 


3. O 




2:D'' : 


3. □ 


Vb. 


2. □ 


3, □ 


4- □ 




.3. □ 


4.J3* 


2. 'd '• 


3;.b" 


4. 


2. □•' 


3. □ 


4'. □ 


2. □ 


3. □ " 


"4: □ 


2- □ . 


id 4 


4. □ 


2. □ 


3. □ * 


4..D 


2. □_ 


^3.;D 


4.cr . 



•f 



' 1 



43 



(15) Has there been any change In your ability to do j ob-related skills 
. and activities (like typing, using Joolsand machinery, assembly 
'Work, serving people, and so on) durihg'the six months since you 
left the rehabilitation program? (Please check one) 
, □ 2. No, there has been no change. V 

□ 9- I don't know. , ; , 

□ 1. Yesl, there hasibeen a change. 

\ A, If yes, please list the skills or activities below, art? 
check whether you have improved or gotten worse 
. . in ft N . 

the last 6 . , Has stayed " , Has . Has gotten 

m 99. t ^Sr* n y a b«)ity to: » the same improved worse- 

/ — ■ — ' □ □ □ 

• * — □ ■ □ □ 

— r □ □ □ 



(16) How satisfied are you now with the information provided by your 
counselor about your disability and the opportunities for 

» improving your situation? (Please check one) 
. □ 1. Very satisfied 0 * , ~ 

□ 2. Somewhat satisfied 
Di,,3. Somewhat dissatisfied 

□ -4, Very dissatisfied , * 
v □ 5. Nc^opinion 

j ♦ . . • I. r ^ 

(17) How satisfied are you now with your counselor's willingness to 
listen to your Ideas and 'suggestions when, developing, your, 
rehabilitation plan and^ob goal? (Please cheekbone) * 

□ t. Very satisfied 

2> .Somewhat satisfied V% o 

□ 3. Somewhat dissatisfied ' • 

<- □ ' 4. -Very dfssatisfied # 

□ 5, No opinion : f • • 

• > * v s 

(18) How^atisfled are you now with your counselor 1 * promptness in 
providing^ervlces to you? (Please check one) 9 

□ 1. Very satisfied '< ' ' - - 

□ : 2. $omewha£s*atisfied < . * 1 

3*. Somewhat dissatisfied *' >' 

□ 4 -Very .dissatisfied- - c 
; "□ 5. No opinion/. 

(19) Did your counselor arrange for you. to have physical restoration 
services (such as medical treatment, physical therapy, artifical 
limbs, eyeglasses; dentures, , hearing aids, .etc.)? (Please * 
check one) : "Vv ,'• ■ • ,. ' ' 
D\2. - No \~.&< \ ' . • 1 ° • J 




1 ^ r * 



.X 




63 



If ~yes, ho*# satisfied are you with these services? 
(Please check one) . w % \ 

n 

D 

□ 
□ 



*1. Verysatisfied^ 
Somewhat satisfied 

3. Somewhat.dissatisfied 

4. Very; dissatisfied 

5. . No opinion , ^ 



* - 

wfrV 

Bin? , - 



.5 



64 (20) 



65 



□ 
□ 



9. 
1. 



Did your counselor arrange for you \q have job-training ? 

(Please check one)' * # ^ *- 

□ z <ho ; * ; #> . 

J don't remember. 

.Yes. 

A. 1f yes, h'ow. # satisfied,.are yoa .^ith_the kind Of 
* training you received? (Please check one) _ 
1. Vqry satisfied 

□ 2. Somewhat satisfied ^ 

□ 3. Somewhat dissatisfied 
D 4. Very dissatisfied 

5, N.o Opinion * * ' * 



y 



□ . 
□ 



67 



s 



rater/ v 



66 (21) Did your counselor help, you look for a job? (Please check: one) 
K D 2. . Np^ 

9. l : dorVt remember* *• r 

1.. Yes " : "* - 

t A. Ifyes, how satisfied are you with this help?^P/ease 
. * • check one) <. : • 
-*<ti 1. Very satisfied • . 4 ^ 
Ch*2. Somewhat satisfied 
□ 3, Somewhat dissatisfied % * 
m \ 1 * □ 4. Very dissatisfied ~ * \ 
\* O 5*. NoopinionC . 

68 (22) How UifilUl were the services or training you received from tt?e 
rehabilitation program In helping you to get your present 
: situation? "(Please^ check pne): 



□ 

a 

D 

□: 

q 



1, 

2, 
'3. 
4.- 

8,. 




The services on training were of great use. 
The services or training were of sprite use. 
Thfe' seryide^ or training were of no use at all. ; 
\ have no ppfpjoh: . • , 

I r§cejved no services or training from {he rehabilitation ' 1 

// / A; \ * ^ ' •. 

69 (23) How useful were the services or training you recefyed from«t^e 
rehaibiHtatlbh-prograni in helping you to perform In your preseUt 
J . situation^ ^yi^^^yifitA'oi little or-nb use in helping youVgetf ^ 

. . l&r((8Jea$^ { \ T^* : - . , 

" iv Tlfe. services (b"r -t(aining r were of great use. * ^ 
2;V Th^ servfces .6r training w^re of some use. 
3. >Tfhe.Y?^'^^^ ra .'^PS V^te °f no use at all. ; 
I have no*'olpfhio'n: ' • # / 

I received ho\services.pr training from theVehabilitatioa 
program^ r ^ . . 



/J 



As , 



'70 (24) How satisfied are /ou with your overall experience with the 

rehabilitation program? (Please" check, one) 
/ CP T. Very satisfied- * . □ 4. Very dissatisfied.. 

O 2. Sortiewhat satisfied □ 5. No opinion 

□ 3. . Somewhat dissatisfied * . , 

* N. 

71 (25) Would you recommend yourvocationaUehabilitation agency \6 
a disabted friend? (Please check one) * * 
' .□ t Yes.;D 27 No □ 3. No opinion 

^72 (26) Wfib answered this questionnaire?' (Please check one afcd ///ft 
in the blank) * - 

□ 1. I. the forme> rehabilitation program client, answered^ 

this questionnaire by myself.' , 
' □27 lV the former" rehabilitation program" client, answered 
- , ' this questionnaire with 4he help of another person. 
* ' * • - ' ~ 

- ' This person is my: {Please give relationship) . 




P 3. 



" 



Another person answered this questionnaire, for me. 
This other person is my: (Please ^give relationship) 



*^s> : " 

my--- 

* V 

fifes. - * * s 

WW 
wm.,^< ■■-•< 

Wf^ ■ - 



(27) Do youAave any comments yo.uiwlsh to make about your case 
« or tKf program, or ls^tnere anything else you thlnk .lt would be 



V 



'A 



1.1 



> \\ • . ; • - • - 


V ^ — " ' .. ' ^ * ~ : I 


. * - >- » v - . ^ v . — JL_ 








<* J o * ? — P ^. t ■ 7 . - . 

5 73-7^ (28) flow long t0 fHI out this, questiqrtnaire? 

\ fc '/ ' - * M - /Minutes K f # * Ui ' ^ 
(29Y Do you wlsh to be contict^d ; &V^ 

Agency for 'fii^^r *Ver^<^^f^. Pl*ase provider, ~ 

* i " w ' . * - * */ ^ ,'\ • * f ' « ^ 


* * K ' • , . ' \ ' • 




: \ **• / ■•• ■ ^ ■ "... •■■ y.-;; 


'r .V ■ — ' : 




Sjate Address 




• StafeV ' »'--£■ ' '* r ^ { P* 


'\\n' s . " - * ^ /'Area Code ' ^ ^ / 


TetephonA Number 1 * 




- n 







, : . . ' SIX-MONTH CLOSURE SURVEY * 

*• • REHABS FOR. JANUARY, FEBRUARY, MARCH/FISCAL 



YEAR 



+ 



1980 

V 



"^i 



Month- 



Samp le- 
^ize 



<Not .... 
DeliQllfett 



Nat 



Returned 
•Initial » 
.Distri- 
bution 



Returned 4 
Fbl low-Up^ 
Letter • 



Returned 
Follow-Up 
.^Question-? 
naire 



Returned 
Follpw-Up 
Phone Call 



Total 

Usable 

Returns 



Percent 



July 



>28- 
24 

52 



1 
0 



27 

2.4 

51 



8 . 
10'" 

18 



' 3.- 
0-. 

'3 



'•'5 
5 

10 



3 
1 



r 



is 

16: 
35 



70.4 
66. r 

68.6 



August 



52 

.107 



2 
6 



50 

'99 



14 
8 

22 



• 5" 
.10. 

15 



'6. 
7 

13 



5 

' 4 

•9 



-3S9 

. 29 
59 



60.0 

. 59".~2 

59-6 



September- 



49 

V .53 

^102' 



3 

.3 



46 
50 

■96 



•7 
16. 

23 



. 5 
.10 

\ 15 



11 

9 * 

'20 



24 
37 

61 



52.2 
74.6" 

63.5 



Total 



129 : 
132: 



.26,1 



. 15 



123: 



24*6 



2*9 

'.U-,34 - 



631* ' 



13 

,20. 



33 



22 
21 



43 



16 



73 
82 



155 



59.4"' ? 
66 .7 ' 

.63 ..0 ''. 




i^MWCS^ W 1 ^?^. ' : & ^SIX^MONT^ 



(1) Which of- the 'following statements best describes 
• • your «p.resent situation? 

I ea^n a wage or salary at a regular job." 1 
1} earn a wage or salary, in a Sheltered 

Workshop < ' • 
I am Self-Employed ^ 

I earn a wage or salary' in a Stafie-Managed " 
. Business Enterprise Program,, 
I am a Homemaker. 

I work on a a' Family Farm ©r in a Family 

Business without pay. * ' 

1 am not working at present - Student. 
I am not working at present- - "Retired . • 
I am not working at present - Trainee. - > 
Other ... *N ~ 

My situation/ is not listed above. 



1. 
2. 

3. 
4. 

5. 
6. 

7. 

8. 

9. 
10. 
•11. 



TOTAL 

151 
64 

- «"*!• 

I 



_ 1 
28 

• 1 
0 
13 
2 
22 
12 



• PERCENT OF TOTAL 



TOTAL 



PERCENT OF TOTAL 



p.. 



(2) How satisfied are you with your present situation? 



A * 



1. Very satisfied. ^ ' 

2. Somewhat satisfied.. 

3. Somewhat dissatisfied. 
4'. Very dissatisfied.' 

5. No opinioti. • • ■ 



\ 



>.f- 
« 



143 

46 
47 
17 
21 
12 



92.3 

-32.2 
32.9 
11.9 
14.7 
8.4 



i 



(3) 



About how many hours a" 



week do you work? 



!• I usually work . 

Average holirs worked 
2 . JL don 1 1 know . * 
3; I^am.not working. 



36.9 



' 1* 

TOTAL 

142 
78 

. 7 
57 




What' are your present .earnings^ 

1;{ X earn * - [ • 

'I ; /Average dollars - $16l.£|5/* 

'^;.^^d6^ r t; J know ■ "1 ' \. . V 



t 1 * 



• 



e a -wage 



TOTAL 

136 

.. .\69 



:5- 
56 



PERCENT OF TOTAL 



91.6 



54.9 
4.9 

4oa 



PERCENT OF TOTAL 

■ 87.7 
50.7. 

. ' ' * :•, 3.7 
"> .41.2. 




I- 



HowllorigLRave.-.you _been. in._your" present _ job 1 




had th>s job" 
4 Average months 49 
m I don't know 
3 • I~ am /iio.t working . 



(6)^ If you usually work, have you^een ou^ of 

\ work at any time during the last si* (6) months? 

1* No, I have not been out of .work at any time. 
. \ 2 f ; I don f {; remembefr. 

3. Yes* -I have been out of work.. /f 
, Average weeks * 21 v 
.4. I dpn't usually work. « r 



J 

TOTAL 


PERCENT OF TOTAL 


? 140 




- 79 
3 

58-- .. 

total 


56.4 . 

I 

2.1 
. 41.4 

PERCENT OF^TOTAL - 


129 


83:2 


62 


48 . 1 


5v 


v 3.9 


23 


.17.8.- 


39 . 


.30.2* 



fl) 



• . •' 
JET.. •*-•'•• 

ftp ' 



■What^ts^yout largest source of, support? 

* • / \ 

1* Current earnings from a job.. 

2. Earnings from ^nteres.t,^lividends, or 

. rent payments _ , 
3* I&miiy and Friends \ * \ 

* 4 . "Private relief agency 

' 5. SSI - Blind \ 
< 6.' SSI ^ Agea * 

7* SSI - Disabled , 
. 8. Aid to families with dependent children 

* 9 • .General . Assistance ^ 
10." Other Public Welfare 

. ll.^Public ^Institution - '\ 

■l^^orker's Benefits' 
. 13. SSDI • - *s#~ 

'14* Veteran's Benefits 
a 15 . Private . Insurance 

16. Other - 



TOTAL 

152 

' 66 




•L 



jr. • 



income, if any, did you receive las ;f 




TOTAL 



140' 
'■ 26 



"A! 



PERCENT OF TOTAL 

' 98.l ' 
43.4 

0.0 
10.5 
0 

0.7 
5^3 
5.9' 



2.6 •**> 
. 6.6 ■ 
3.'9 
0.0 
1.3. 
8.6 
2\0-\ 
0 VI .\ 
8,6 



- PE^bENT OF TOTAL 



105 
9 





totak: 



i!ow;inuch drd^ypu, v earn dtiting^the last 6 month*? > 145 




1*. I earned' ^ , ** » m 

Ayerage Amount $3,18$. 85 ' J - 
2;* i'did not' ear\i an^ttiin^ during the la£t 
4 . ' * six months.- s A * 0 % * 
I don't remember • 1 



v 



70, 



50. 
*~ 25 




TOTAL 



PERCENT OF TOTAL 



ft.. 



(10) When you .finished rehabilitation services, was 

'•anyone"*£n your family sable to work of work longer . 
.< hours iat a job. they already had because you were - 
able to care for yourself? - ~ 

H > 

2. :Np. 

9.* #t4on*t"know. ' v' • " - 

• ■:■ 1. Yes. . V .' 

," . ■ • A. If yes, is this%till the case? • . * 

1 1>\ Yes! _ - > .. . . 

"" 2. No. r » ; ' 

. "*. 9. -'I don't know.- ' 



117 

94 
■ 9 
14 
11 
10 
1- 
0 



^ TOT^ j p 



tllfeC Hpw^do iyou feel* about yourself compared' to 

Six months ago? • j . ' r \ 

1# I*£eel the same about myself > * • ' 

2v* X f;eel better &boutSnyself . * y mt 
''3* « $ feel 'worse about my$$lf . ' / * ' : ' 



TOTAL, 



75.5 

80.3 
^.7 

78.6 
90 -.9 
9.1. 
• 0.0 



.•V 



144 

68 
42 

18 '• 
. 16' 



PERCENT OF TOTAL .» 



92.9 





4.7.2 
29.2 
^12.5 
11.1 



ST OF TOTAL- 



90.3 





TOTAL 



," V 



PERCENT OF TOTAL. 



1 



Wife 

' Slicing; . 
Ifll'lS^ '.Has'-s fayed the same> * ' . 

Ipl!!^ V." r 



404 



MS^M^^^i^k- £he JPhorie 

ft^^' v Has; : improved* . - 
^/^^•'ttaf "gotten wors,e.., * 
l^^i^jlote'^uf e*« , ~~~*^ — . 
l^^^o^iti g oh the Street ' ^ 
f|p&^f ' Has 4 stayed the same. 
&V V "2 4 s ' * Has improved. 
j^''3> *Has%otten worse. ' 
&0 r 4. Not sure. 

^ . Using ffublic Transportation 
W'> 1. Has stayed^ the S"ame. 
jg< 2. Has improved. 
§|$f*'3« gotten. worse. 

fS^'/A* Hot/gure. 
^ v *£i ' Driving. a] Gar ' • 

Has /.stayed the samew 

Has improved.* _ 
lltv.3*. Bag gotten worse. ' . 



a. 



VNot sure. 



W« ■ 

pi 5 - .' -H r : 



.115 


- *74.2 


98 


85 . -2' ; 


"^^14 


. 12.2 


■ ir 


0.9 


2 — 


1.7" 


-113' 


72.9 


92 • 


81.4 ^ • 


16 


14.2 


' 4 


3.5 ■: 




0.9 


'iss»- 116 * 


, - . 74.8 ■ *. 


#'-92 : 


.79.3 


14 


. 12.1 


8 


6.9 


• 2 ' - 


1.7 , 


v 109 ' 


• -7Q.3 


89 s 


81i7 * 


10 


9.2 


5 


4 . 6 *^** m *' ,* 


5 


4'.6 , 


105 


*' 67.7 - 


"85 


81.0 * 


in 




2 - • 


1 Q ' 

1.9 




3.8 










• 


/ - 



247 





m^M^ ^ouxare primarily a Homemaker, has your 



^l^r 4Q-j^%any of the skills, and activ 
; -\ :|el^ted* to ^Homemaking changed^ during* the 'i 



activities 
past * 




" ' 3.Y^ v Has gotten worse 
- 1 A No tf .sure v 

fll^ - * ■ "l Baring - for Pets 
jfe/ ■ ITf 'H^s. stayed the same . % 
5^5*' - , • ' ;Has ^improved . * 

1 f 3/. -Has 'gotten worse. 
•4 • Not sure-- - 
c. Cooking -andj: ixing meals. 

1 . Has .stayed the same . - 

2. >Has improved, 

3. . Has gotten w6rse. 

4. *Not.sure. *«' 
d» Washing Dishes* 
1. 'Has stayed the same. 

Has improved. . - 
3 r »Has gotten wors*e. * 
4 •* Not sure... ' 

e \ Cleaning 'Hous e » . • 

;TT ;Has stayed £he Vame. K . $ # < 
2$ Has 'improved.: 
3v ;Has, gotten worse. I*-'" 
Jfl "^fpt^sure^ . . - ■ , 

f ^Doing Laundry 
i~. ; Has v stayedvthe same. - % '„ 
2; r V Has improved . ** 
fll^K 3?' ^.ats ^o^tten wdrse . * \ >" • 
M^'*\;|f4« «otj'suire^ . 

vg.^ iSKdpp^ng-fo^Food and Groceries 
: -l* ^'Has stayed the same • # 



i 



fir-* : 

If: • ■ 



i «ot sure, 

fe>,W - ...... . - 

3821- " * ' f s ' Ha% ;imprayed . 

' ^unnin^ Errands ^ y . 
P^;V;- .1^ ;Has: :s t ayed ' the same' 
k^^r ^-.^v2>v .Has; amgroyed. 




56 

43 , 

7 

1 

5 
57' 
47 

6. 

• 1 
3. 

n 

54 ' 

69' 
5J 
9 

'4 

• 1 

'47 
12 
8 

2< 
70 

12 

' 7 • 
- 1 
71' 
43- 

4- 
67* 
46~ 
1*6-' 

2 

'i 
67 
45. 
' 12 . 
•8 
2 

62? 
'42 
5 

. 7 
8 




36.1 
76.8 
12.5 
1.8 
'/, 8.9 
- 36.8 
82.5' 
' .10.5 
' 1.8 
5.3 
47.1 
74-.Of 
'15.1 
8.2 
. 2'. 7 
44.5 
73.. I . 
■ 13.0 
5.8 
. L.4- 

' 44 - 5 
68:1 

17.4 . 

11.6 

, 2.9 

45.2 

. 71.4 

- 17.1^ 

' io.(\ 

1.4 " 

45.8 . 

60.6 

25.4 
••8.5 V 
- 5.6 

43.2 

•68.7 



23.9 
. 3.0 
4.5 
43. 2 
67.2 
i7'.9 
11.9 
3.0 
40.0 
67.7 
' 8U 
It. 3 
12.9 



^4 




1 ;1* ""ZT 



TOTAL 



PERCENT OF TOTAL 



fp|£I5;)- ^PSs'^ere been any change in a your/ ability to do 
*" - '^job-relateci skills and activities during, the>£ 
six ; .months since you left the rehabilitation 
program? * 
*2. No there ha£ been no # .change#* 




9* 
1. 



mi- -. 



I, don. t know. 
Yes, there 'has been a change V 
A. If ye^, piease list the skills or 

activities below, and^ch^pk whether 
* .you have improved or gotten worse dn 
\* * them: . ' . 



Give injections. 
Typing , 
Working* skills 
Serving people 
Inventory control 
Hearing 
Filing 

Work in tool room 
Read and write 
Sewing 
Walking 
Do close work 
Yard #ork v . „ 
Talking * • 

Housework 
W6rk^|ast 
'Programming 
Using tools 
* Total 



•114 
84 
9 

j 21" 



13.5 
73.7 

r.9 

18.4' 



Same 
0 • 
0 

' l", 

0 

0 
-0 

5 

0 

T o 

• :V .D 

■i 

i 

o • 



' Improved 
1, 
• 1 ' 

' ••" 0* 
3 
1 
1 
'0 
V 
- 0 

1 

• < 1 

* 3 
1 

1'- 
. 1 
1 

' 19' 



■ ■■>*,. - - 



/ 



TOTAL 



|P?&S)' ^ow satis fied arfe^ou now with the information 
provided by your counselor about your di£-. ; 
^rr&Zl&y an* the opportunities for improving- ; 
^ 'your situation? ■ 





"PERCENT OF T0T£L 



130 •• 


*•*»• ' 83.9 


67 * 


51.5 


24 • • 


' IS. 5 


- 10 


. 7.7 A 




3.8 


24 


V 18.5' 




^^Ml^^^^^^^M^'^^^ yoii 'now with- your coun 
* 17* ^i^|^8 n ? s s , M Xisten to your^deas and 



PERCENT OF TOTAL 



counselor f s 



Suggestions when developing your Rehabilitation 



Rlan and Job Goal?^ 

fJ^v; ; '' ".'Very satisfied. . A 
2v\-S6mewKat 'satisfied. 



hp 



^Somewhat dissatisfied. 
4. ^Very- dissatisfied. 
No' opinion. -V 



5 



' (18) 



.How satisfied are you now w^th your -counselor f < 
promptness m providing services to you* 

1. Very satisfied.. - \ * 
' 2> Somewhat' satisfied * 

3. Somewhat dissatisfied. , 

4>. * .Very dissatisfied. . * 4 ' • 

5/ No .opinion • 



r- 



l^^l?)*; Did' ? your counselor arrange for you' tp\ have 
v . . Physical. Restoration .services? * 
2.—No. : v ' ' 

. * v :% ' I| don 1 1 remember . 

/Yes. . > *>- • ■ 

P^SV'' A* yes, f /how satisfied are you with * the 
v< ^\;seryice6? • . . . 

-•Very* satisfied.' 




S®B?^' V >5 • * No opinion/ , - . 




mm 




120 


. • . 77.4 ■ 


65 


° 54.2 




,14.2 • 


r 
0 


5.0 . v 


4 


.3.3 


' 28 


23 .3 


*\ 




■>» * 


- ' '- ,V / ( 

** 

* i 


TOTAL • 


. PERCEHT Of TOTAL . 

\ " / 

80.6. 
v • . 64.0'_^ 
14.4 . 


" -425* f 
-80 

- <8 


; '6 

;> 8 

. 13 


4.8 
6.4 
. l°-4 


^ ' ^ 

TOTAL* 


, PERCENT OF TOTAL 


* 137 . 


■3? 

• 88.4 


46- 


* *< 33.6 


5 - * 


■■ 3.6 


87 

> 


. * 63.5 


84 


■- 96.6 


. 65 * 


' . - 77 M. ' 
' ' - * ■ ,13 A" 


"•' 11 


; .4 . 


•'4.8 


2 


..2.4, , 


2 • 


2i4 





C29)i3Pii -ybur counselor! arrange for you. to have, 

^/;NoV- ■ ' ' ■ \ ' 

-p. 'Sf>*^T^dbn* t remember* " 
.Yes* - " r 

A; 4 -Tf. yes, Vow satisfied are you with the 
v of .training you received? " . »■ 
, 1. V£ry ^satisfied." - '* v * 

2i Somewhat satisfied. - v r • 

• 3. . Somewhat dissatisfied, 
4. ;Very dissatisfied. . • - 

'£»*'il6 opinion: I* . 



kind 



■ TOTAL 

13H 
103 

19 , 

02. 
. 10 
5 

. 1 
„•? 0 
3 



PERCENT 0/ TOTAL 

79.2 
. 6.2 
14.4 




(21)''«$id .your cpuniselor help you look »for a job? ^ 

2. No. . 4 ' ■ 

9. I donJt remember . - , 

1. Yes. A - 6 

\ A. If yes, how satisfied are you with this Tlelp? 

1. Very satisfied. 

2. Somewhat satisfied. 
3; .Somewhat dissatisfied. 
A» Very dissatisfied. 

J|- 5. No opinion. 



TOTAL PERCENT OF TOTAL 

125 ^ 80.6 

95 76.0 

5 4.0 

25 * -20.0 

23 ' . 92.0 

12 * ' 52.2 

7 . ' 30.4 

2 8.7 

' 1 * 4.3 

.1^ ' 4.3 



fe'r:- 



(22) 



m 



WSJ" 



i 



* • • * 

How useful vere the services, or training youp 

received from the rehabilitation pr>ogrant 
* in helping you to get your present situation? 
1. Services or tpaxping were of great, use. 
2.,** Services or training were of some us£.. - vp 

3. ^Services or training w|re- of no use. 

4. ' No opinion— . ■ < ; , ,. 
,8.*" I recievecf no services or training. 



tap 



fe'-:(23> ^HbwWful were the slices or tr^in g ;you 



TOTAL. 1 



If ' recpiv.ed from the rehab, program, i#fielping 

youl^etfornf^B your present situation? 
MM^t&^^m sepyiW or graining were of. great use. 
»>->^ a?^".^.; ^^..^..i^,, training were of some use. 

training were' of ^rio use.. * 





o TOTAL 


. PERCENT OF TOTAL 


120 


. " 77.4/ 


'• T30 


25 ;o; 


• '26 * - 


. • 21.7? 


15 


' . ' 12.5/, 


> 10- 


' • 8.3 


, 39 


32.5 . 



feERCENTTOg TOT^L 




555 J. 



'3^ ,* 



mi^^^MMPMW-^- you:"4th your overall . • 
W;',/:'*- expe#ence with-the rehabilitation program? 

^Ff : *i''^' Somewhat satisfied..'". < ' 

; **, Somewhat dissatisfied.. ' • 

V'*-' Very .dissatisfied; *'• 




5. No opinion. 



■TOTAL - 
/ 135 

r 79 

" 23 

"9! 
11 
13 



PERCENT OF TOTAL 



87.1 • v *' 
•58.5. "Y 
17.0 ' . \ 
6.f v * . 

V 



9.6 . 



||v (25). Would you recommend your Vocational 



It 



Rehabilitation Agency^ to a disabled friend? 
.1. Yes. 



*No. 



•3. No opinion^.-. 



■ rTOTAL 

139 
110 

.. 12 
17 



PERCENT OF TOTAL- « 



> 89.7 fA\ 
•*79.1 'V y] 
.8.6'< \ 



12>2' . 



we 



{26) Who answered the Questionnaire? " 

1. • Client. '," 
/ % 2* Client with help of another person. 
r - 3/ Another person. 



M$v : C27;) 9 Do you- have any comments? 



ItfifK 
p$i% ; . .. 



X. Yes;.' 
.2,. No,. 



W?£t$$) V H0S?'*k«lI, did it.'tak^.jro'u -tc 

S^r^^^^Q^^PWaire,?' ? * . ' 



J:o fill out this 



, TOTAL. 

- 142 
* 11-2 
' ♦. 22- 
. 8 



'• PERCENT" OF 'TOTAL , 1 • 
..78-9 

• " 15.5 ; ^ - 

. •'• ..5.6 




^fe^l^^Sfc' Co"^ contacted?; ' ' * 



TOTAB 

* 81 
'74 



.TOTAL 



■ 15.8 



TOTAL 
155 

• 52 

:■ 103r 




• ^PERCENT OF TOtSl '* 

'"«*'' ' '52^3 
. , . ■ *?»7f • . 



r ■;• > 

PERCENT , OF TOTAL 

» o.o' 



PERCENT OF TOTAL 

• 100 --° 
. , ' ' ' . 33;5 
'66.5 



1 < 




APPENDIX 13 

,..„,.. FIELD TEST OF THE FACILITIES 

«*'.*' ' *" - ' INFORMATION SYSTEM _ ; rf 

PURPOSE - The overall purpose of this project was. to develop a Facilities Information 
ICy^JSystem (Fil.S.) that would be designed to meet the information needs of Facilities, , 
Sjtate Agencies, ' and t;he Federal Government. 

p ; \' . " ' * . • ' " • * 

Ife?" "OBJECTIVES - The objectives of the field .test are: I \ . 

Evaluate and.'refine the*F„I.S. . , w . V 

- 2. •Ensure that a nationally implemented Ma^gemeni^ Information System- is qff 
, value to each user, • 

§lw' ? > * 3. Recombend any refinements and/or .revisions to th'erF.I.S. 

4* Recommend whether the system should be implemented nationally, — 

.- * * * 9 

> ; , v- * * 

f • * * . . * 

Our field testing has identified certain areas of concern for consideration, 

:> y . • ■ - 

These are listed below in categories, . * - > 

SELECTION - Selection of participating facilities should be mutual,, The State 

£X>. : >' ] r \ 

- Agency and the State Facility Association should decide who they want to participate. 

Jfio - h ' - # 

fr?' : ^owever, r the State Agency should have the final decision. • ' . * 

- - * ** - ' * - ■ • * - <* 

' CONTACT -.The method of contacting the facilities is yery important. In .Pennsylvania/. 
The t Pennsylvania Association of Rehabilitation Facilities (PARF) contacted their 

member facilities.' The Bureau contacted the ribn-PARF, members. > *' 

'■ » "\. ■ * ; • • , • ~ • 

p*; TRAINING - T?he training for the p&rticipants-^was conducted in two sessions. * The first v 
sessioft was held in Harrisbur.g in May 1980. The Second 'session was hel3 in Pittsburgh • 
' ' in June .1980 . ^The sessions -were jte\l jpliinried* and presented. 

However, . there was some resistance on the part of the facilities. „ This may have been 

: bei^tlse : • ' * * ■ /- ; ^ - % ' " * - * : s 

P^^^^-l.: *!{; vas r^ot made clear from whom they were t<5 receive Technical Assistance/ 



H 2. "The^e v^as too n much' material ^presented in too shprt a time. 



SSC ' -It may have been better received if individual sessions k had been conducted. We 
^S'' ^realize, tthat this method 'wouli not have been economically feasible. 

FIELD TEST - The field test started July 1, 1980. However, since 0MB approval cff the 



Sfe, Y , forms ^as not ^received, Pennsylvania/Model Evaluation Unj,f (PA/MEU) decided no.t to *, 4 




■ V 




K ..• **** decided t0 c ° llect the data for Pennsylvania. After data collection .was* com- 



Kr*' .Plated f # PA/MBO was to receive the data for analysis*. Howevet^we'have not received- 
It is our understanding that PARF was' having, trouble obtaining the data. We* wet e 



the data in time for it to be included in this report 

ft; . 

|T, informed • that we would receive aggragate data instead of. -the actual forms. 



V CONCERNS - Since PA/MEU decided not to conduct the field test^FARF selected additional 
^ facilities to 'participate- These additi.onal facilities were selected because there ' 
^ were not enough participants. ^They were selected with our knowledge ibut without our „ 

f? * ' * * * he- * 

consent, i * * ' / 

There was some concern about the 90-Say follow-up by the Medical Facilities. This 
concern t&p based qn the fact that, once a client, leaves the medical facility, the/ 
v< - facility- does not maintain contact. «> m ' - 

' • ' . >• v . . u c 

V ' There was also concern about the validity of«the data collected. The 'smaller 

'c ^ , < * • 

^facilities probably do not keep tfie t£pe of informatiorjjfceing asked for^. This may be 



T>ec^use: 

If; - i 

ft- "■' * * ' * . '' \ 



The VR client population is smalT'compared to total, population. 
2. §taff requirements are not sufficient to allow* for the„ collection- of the* data. 



5k » k s 



\\ .3. The overall objective shotild have been* crystalized with a perspective for 

1IL . . > * • ✓ * *• ■ . - 

l * • meaningful utilization for translation to the statg^. • * \ 

p MCOMMEftDATIONS - If it is .implemented, it *shotftd be revisecj to include the above-list^ 
f^?-' cbnp^rns/ It is felt that with; the date collected from th'e field testing, an analysis 
could be made^thaft would result in appropriate, modifications f or % meaningful iriforma-* 

tioti system for * the. spates. - * • .« <■ ' * 1 ''^'/^ 

: _ , « »- . ■ ^ 1 „ f • " / 

feXv> > , ■ . •* * 4 • ' 



108 • 




254 



4f.' 
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TASK VI. DEVELOPMENT OP NEW EVALUATION CAPACITY 



■ PRESENT PROGRAM . 


: . : — v : ^ 

;:. • PLANS FOR INCREASED CAPACITY 


— — " V 

Case Review' Studies- ' - • - 

**' . • ' r 

- \\ - * • • \ v - \ ■ 
. , « . • . ■ > - 'Y" ; ' 


■ - < , 4 / 
3U Con^wt further reliability studies ^ 

c K Kei xne uuideilnes tor Keview Statuses \ 

Refinement of Case llevi'evrSKeets * - 

4. Refinement of the Dissemination and Utilisation Process 

5. - Establish Additional -Studies • 
\ . 6. Increase Base-Line Data • 

. 7.- Annml Plan foi; Case Reviev;s. 


Profile Malsrsis.techiii^if^MlT^ ' 

* * ' - * w \ ' » 
• • • ' . * v ' ' * *' 
' * \ * • ' ' 

• ' • m • ' ^* 

• 1 ' .'; • - . *, ' .. " " .. + ; 


"1. Refine and .Increase Output Variables - 
'Aaaitagn oi, Jnput and Process Veiriables 
3. Addition of Predictor Variables 

Consultation oft Statistical-Aspects of) Profiles 
K^jcpana otai4 irainmg m Profile utilization 


•'.,.*•-•• - ■ « . • 
-Management . Information System - - 

; " ' ' * -' . . ' . '• . .V- ' . •* 

r * * 

: , 1 ^ -a c^-l: — . — — , — 


" 1. ■rorine'* and, Update Information Available to Management 
2, „ Develdp Procedures* for Information Requests 


Evaluation Section Organization 


Refinement to^Achieve a-Nev/ Model N - Vt 

2. Define Evaluation Sec.tionTGoals and Objectives 

3. Show Functional Interrelationships with CO, R0,.and EC Staff 


Follov/^up Studies \- y " 


f' l t Standardization a^d CrossrValidatio^ w - 

y -2* -Cross Tabulation I 0 

3- Format- Data for Disseminatibn 
• 4.' RcvisicQ of, follow-up , SuBvej- Form v 



> 

H 

X- 

M " 
4r 
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TASK VI. DEVELOPMENT OF. NEW EVALUATION ' CAPACITY , " 



NEW PROGRAMS 



' * — 1 — — *- ■ 

: 7 NEW PROGRAM . ^ 


. ■ * # ' WORK STATEMENT - - " ' ' ' * 


Case Review i>roce N ss 

> * 


1. Correlation of Delivery, of Service to Rehabilitation Success \ 

2. Analysis of process Indicators \ * . 

3. Utilization of Annual Plan for . Case Reviews 

; 4. h Respond to- Requests from State Board, * Director and Oth^r Central Office 

Staff- fqr\Specific $ase Reviews* . - ' 


Similar benefits 

■ ^, * 


1. ' Identification of /'Similar Benefits" for data gathering purposes 
- 2. Analysis of ,Use 'of Similar Benefits _ 
; 3. Computerization » 
4. Dissemination of Fdfmatted Data on Similar Benefits Usaere 


Client Satisfaction Studies 
Irfer -rr7 * % / 


— t T— = : => — r ^ = _ 

1. ^Relationship Between Client Services .and Client Satisfaction 

/ 2. ' Relatipnship . Between' Client Life Satisfaction Index ahd^Client Satisfaction 
* * with Rehabilitation Services * 

3. Validation*' of • Questionnaire, Data* . ; ' . *r 


Facilities. Data Base / 


1. Establish Forihat' for ^Interpretation of Data B£se ' 


:|l^|/iMethodg • ' • • 
'.Technical Assistance Center ; 

^Ap^prrr. 'V — — ; . 

^y|lu^tip^Conp6nent for 
^^lint^'j^sistanpie Projects 

^^f^ggfa^ Section ^ 
• ^ f^lbledjli^, PqpuI atioh 


Develop and Initia^d^eYfe^tiye needs assessment and dissemination of material 

„ Initiate and Implement Collection, Analysis, and Dissemination of Data to . ' 
Determine Achievement of Goals and Objectives of Prpgram . . , o - 

Define Goals, Objects and Functional interrelationships of Evaluation Section, 

Analysis of Categories of Severely Disabled for "Dissemination " - 





test ' . 

3x3 



,25-8 • 
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life • ':' •: ' *>. ' PROGRAM EVALUATION SECTION' LIBRARY . Appendix 16- 



^ The Program Evaluation Sectipfl Library was established as; the* result 

f^v'/' of our receiving this federal conf^c^.wIjLch mandated ,that a libraxpr' dealing 

1 1 ' " ' program' dvaluation.be developed. ^ ^ , ' m * 

jg^V ' - " In the. beginning aS^that .was envisioned by Pennsylvania's Model J3valu- 
.afcion Unit was a small library dealing solely with evaluation subject matter, 



However, as th^work* progressed it became apparent that there was a need' for 



v 



ST- 



1S> - 4 , ■ \ ' - . . : * * 

a more extensive libbary which would serv^ the entire Bureau and the^library 

that wa|' established, while known as the. Progtf^m Evaluation Section Library, 

is, in feet, a BVR library 7 As 'a result-, the. Program Evaluation Section - # , % 

Library has approximately £,100 holdings covering many areas of vocational , 

rehabilitation, including new rehabilitation trends, current legislation, 0 ^ 

- specific liabilities, assistive devices,, etc v This library is housed, in / * 

the Central Office of the Pennsylvania BVR/l6 serves JBVR's 15 district 

Ifev/- "» offices, 4* regional off ices * the Hiram G.- Andrews Center (a 500^ client, - 

rehabilitation center operated by BVR) and 3 Disability Determination Offices. 



It 



||V , . - - * T he creation and -development of the Library required careful planning 



ffrVW * * and much- re sear chr on the, part of the Pr'ogram Evaluation Section stdff. The t 

* .'followiiig is thte i methodology used to establish the Program Evaluation Section 



4- 



pr\-. , Library^ ". ^ , - 7 



£5i 



• Since^his vas a totally new undertaking in an unknot are^. there wa| 
'an obvious need for expert advice and assistance. This was sought from many 
and .varied sources. One of the more* evident needs' was for s fhe expertise of 



professionals in established library systems. The Pennsylvania State 
Sfc '• /• * library ;.-located in Harrisburg, was solicited for technical' assistance on 

thei, establishment, maintenance and continuing development of . a library. In^ 




addition, its Staff provided information on the various activities and- 



^field offices with pertinent -material to meet their needs. For example, . 
they provided- us with a list of all periodicals "received by the Pennsyl- 
vania State Library and placed us on their mailing list to receive notifi-r 
cation of all new acquisitions. In addition, thej^aluation Section 
Library has-been listed in the Pennsylvania State Library's publication 
"Directory of Libraries serving the Government ofthe Commonwealth, of 
V Pennsylvania." This directory ' shows the number and types of information 
resources available to the various agencies, departments, commissions , and 
boards of the Commonwealth of Pennsylvania. 

<» 

We had another very important resource, in the library which is located 

- 

at thte Hiram G. Andrews Center in Johnstown, PA. The Ceni^er Library has a 

section of materials available for use by Center clients and a separate 

section of reference materials for staff use. In addition, itf i&in<the 

' • m 
process of establishing' a medical library for use of' the Center staff. A 

I v 
two-day visit to the' Center produced valuable .information a,s to cataloguing 



lb 



t procedures, suggestions ibout periodicals and books to be Acquired, publishers 
- and other types of resources. - . 

: ' - ' " , • . ' 

We also contacted and received valuable assistance from many sources 

/ * I \ ' ' 

, out side' Pennsylvania. Massachusetts, Minnesota, Ohio*, Oregon fcnd' Virginia, 

\ all haye state vocational rehabilitation agency libraries, the-staffs of 
. which shared, their expeVtijse with us, Two members visited tire Nation^ 
? ; Rehabilitation Information Center* (NARIC) *in Washington, D.C. , to obtain 

its assistence.il> dissemination,' utilization" and iibrary scierifce techniques' 
^'arfd'to discuss k the purpose, of NiJRlC and how it serves rehabilitation pro- 
fessionals. Personal* visits were also made Sto the President f s t Commit£ee^on 
BS^S* Employment* of the .Handicapped, the Office of Handicapped Individuals, the ° - 

1^^,.;,; Off ice, of Civil Rights, arid Project SHARE, all of which provided varioug types 



ff^' 9 f information and assistance • ' Two persdns. attended the first meeting of 
the Rehabilitation Information Round Table in Washington, IKCV,, an organi- 



%L , zation designed to promote the scaring and .dissemination of rehabilitation 



information, land working relationships were established with ICD in New York 
and the. Oklahoma* State University Clearinghouse in. Stillwater, OK. These 



r 

K 

a. 



agencies assist our liBrary in locating material on subjects being researched. 



■ Another phase of the groundwork was contact with publishers, numerous 
rehabilitation -related organizations and other sources that. have available 
!\ and Program Evaluation materials which coulii be'^acquired for inclusion in 

%\ ■ 0 : ; 

• »- « 

the library. 

The actual creation of the library required* many decisions in three 
{; % principal areas — physical needs, acquisition of . materials , and. operations. 

Foremost among the physical needs was housing — a suitable location wher.e / 
people would feel free to visit and browse was selected. Then it' was ^ 



necessary to order bookcases, book covers, magazine racks, a filing cabinet. 



ST\v - and other needed equipment I 

p: . ■ . 

W' x Animpprtant consideration was the selection of a. 'cataloguing system. 

. . . Various cataloguing systems were reserached, numerous thesauruses obtained, 

|v-^. • . .. - # \ . . 

{?- " and the' development of our own system ^considered. In the^end, it was decided 
yf\" to adopt ttfe classification 'designed in 1961 by the Oregon Division of 
f^c, Vocatioual Rehabilitation . .This decision was based primarily on the fact 
|^ f , . that ,it| was a proven system which had beeji adopted with success by various 



^4:"'( agencies which house rehabilitation literature. Providing detail and 



at 



S . the same time allowing for flexibility to meet individual needs, it has 
^|rV proven to.T)e a very usable .system in our library. "Its classification system 
^^'J^dexitifies. nine major subject areas- and t^e'se are subdivided to allow for 



^J^ t j^e^cla$^ication of publications which 'deal with various aspects of these 
I^^tmkfor^sxibj e,ct-s . The major -subject areas^and numbers assigned are: 




100 seracSt— Rehabilitation; '200 series—Disabled and Handicapped Groups; , 
300 seriesr-Impairments; 400 series— RehabilitationlpCooperatipn arid Inte- 

.eration: 500 series— Rehabilitation, Financing; 600 series— Social ^fbblems; 
7O0 .seijies— Government ; 800 ser^es—Administratioj^; arid 900 -series— Ref er- 

'ei^^W*k6 # nof "Related to" the Above Subjects, " < ; 

When a publication is received f v or inclusion in, the library, it is first 

.classified and qataioged. For books, pamphlets', and miscellaneous material, 

two 3 ,f by 5 ft cards are prepared: one by title and one by author,* The 

material is then placed on the shelves or in' the vertical file^, which is a 

file cabinet containing. magazine articles, pamphlets, and other material 

which* .it .is not practical to Store on bookshelves. Material so housed is 

identified in the card catalogue with a "VF" under the classification number, 

A separate card file* wasb-established f orjjperiodicals , of which" we are 

presently receiving approximately 75 (see attachment), , In-order to main? 

tain an accurate up-to-date record of ea'ch periodical received, a card is 

* * . 

prepared for each periodical. As each new i^sue is recieved, it is recorded 
on the cafrd and circulated among the program Evaluation Section staff, ^pon 
completion of circulation', the periodical is re-Kroed to the libn^rylwhere 
it is placed in a holder with other issues of the same periodical. All 
periodicals, are. located together in a section of the,, library which is^ easily 
accessible to anyone wishing to use them, ' ' ' *-"' * 

^ ;: Since the' main purpose of -the Program Evaluation Section Library is. to' 
provide pertinent information to all Bureau of Vocational Rehabilitation 
personnel, a method for dissemination 'of material needed tQ be established* 
TJi£ tool, selected, to -assist in the dissemination of material was- the' "Rehab 
Brief" , a. biweekly .publication prepared by the Rehabilitation Research m 
Institute of the University pf. Florida, located in Gainesville Since all 
..field^ersonnel Receive this publicaticm, it was # decided to include an insert 



^mr;<>%\ v — - . ...... • . % 

with each "Rehab Brief" making its readers aware of Program Evaluation • 
Section Library acquisitions. Each issue of the i'Rehab Brief 11 deals with 
a. specific Subject^ interest to rehabilitation personnel. Therefore, 
/the b^sic format* for the insert is a short one-paragraph synopsis of material 
which is .available on t-hifc subject in the Program Evaluation Section Library. 
, The. remainder, of the two 'page "insert is used to offer to the^fi^ld other 
articles on various subject?, free material they can order from clear ing- 
houses andjpther agencies, ^nd\gny other inf oration we feel would be bene- 
ficial". 



Since the main thrUst-'of the insert^* is to ebtain lfser involvement and*' * 

* * * - ■ * * 
then dissemination of material doWn to the counselor- level, all personnel 

receiving this insert may call the Program Evaluation Section Library \ « 

directly for information. This is an important facet' of t£e Program Evalu- 

ation Sectipn Library since it is a direct linkage with every level of -BVR 

personnel. In .addition, another important asp'ect o*f user involvement is 

, obtaining feedback directly from the field. Field staff are encouraged to » . . 

* * * • 

contact the Library with suggestions for improvement, recommepdations for 
"acquisitions they, feel would be beneficial for inclusion 'In the Program • ; - 

Evaltiatign Section Library and any other .areas they feel would assist *us. 

- ' . * * * 

Technical assistance is an additional service Which the Program Evalu- 

ation. Section Library pr.ovid.es to. persons requesting specific information. 

Thus far technical assistance has been provided on a wide range of subjects, 

a few of which are clothing adaftations for specific disabilities, specialf- 

Szed eating equipment, information on specific disabilities, and. access- . 

ibility .design and functional limitation: To thoroughly research a request, *' . 

all publications pertinent to this^subj-ect which arie part of %he Program 1 ^ 

.Evaluation Section Library are reviewed for possible inclusion. In addition, _ ? 

, if *we dp Jiot have sufficient information^ jbh^ request is researched through . 



\ 



t&fcu- " /;* ( , . - ; * -v- . . , . • - x .. - /' 1 * 

. the- Pennsylvania State Library, NARICf, leseardh and^.Training Centers, 



KRRIs, ind any other source we have knowledge of whefre this material 
might be available. . . ' * 

•\ A loan, service has been generated which makes available to the field 
various publications which can be loaned for a two-week period.* fiach issue 
of the i'Rehab Brief Insert" lists new books available through the loan- 
service. Since it is pot unusual for more than one person to request each 



.publication, a, waiting list has been established and books are loaned on 
a first-come basis, ^Aaoai&pspying each book is a memorandum stating this 
\ publication is being furnished on a loan basis and the due;£ate for it to 
be returned. * , •* 

The'followihg is -\ six -month breakdown of information requested from 
* January 1979 through June' 1981: ... * 



V 



# of "Articles of 
Books 
Distributed 



T 



Jan 7 9 -June 79 
• * July 79-D.ec 79 
. "Jan 80 -June '80 
, July ,80-Dec 80 
Jan 81~June 81 



32 



110 

3^3 •• 
'., 678, - 
1,610 



# of 
Individual 
Requests 

16 . 
26 
72 
73 v 
80 ' . 
. 267 



Average # of « t 
Articles or 
Books Distributed 
per Request • 



2.00 
H.23 
6.20 
4.69 
8.t»2 
6.02 



The useage of the Library has increased dramatically during the past 
year since it has become more. widely kccepted as a source of current infor- 
mation by our staff • We haveVfounff that the most widely requested -infor- 
mation is that which deals with various aspects of placement arid specific 



disabilities , such as mental il] 



Irtjls, alcoholism cancer, heart disease, etc. " 
J The Progr&n Evaluation Section Library has proven to be a useful source 
. information for staff. and a means by which they cah easily .obtain the 
material in, the field: of rehabilitation, 





I • LISTING OF PERIODICALS 



"Accent on? Living" : * 1 * 

"Administrative Management? 1 ' , ,«:•** 

"Aging", . 

"Aids and Appliances Review" ' 
, "Alcohol "Health, and Research World" 
"Alssoan" # , * , 

; / "American Demographics" / " • * - i . 

* f Th ? American Journal of Occupational Therapy" 
^;-u"^iericaff Journal of Public Health" 

^."American Rehabilitation 11 * 

"AmicusVV.3^: \ , ' • ' ■ ' " * 

"Awareness" ' # * 

"Behavioral and Social Sciences Librarian" 

"Breakthrough" 

"Can Do - Will Do" . 
"Caption" 1 
"The Challenge" \ 

"Disabled USA" ' * v ' \ ' . ' ^ 

"Evaluation and Program Planning" * * ' * 

"Evaluation News" 
. "Evaluation Quarterly" 
"Eederal Contracts Opportunities" ' ' - ' 

"Federal Grants Opportunities" . • 

"FEP Guidelines" . 
^/'Findings" . • ' - 

"The Grantsmanship. Center News" * »/ • 

"Guidepost" ' . 
• "Handicapped Americans Reports" * r ; 
"Heatft;h and Fitness Newsletter" 

"Inc." - 

"Independence News" ■ ' * 

"Information Management" 

''The Information Report" * " 

"Informer" ' ' - \^ 

"Innovations" - 

- "Innovator 1 . 1 " - ^ ' \ * 

"Inside Dope" • • " \ ^ 

•"ifiSl- Newsletter" . / - 9 * 

"Journal oif Insurability" * " 6 " * 

^Journal of Rehabilitation Administration" : * 

"Journal of Applied Rehabilitation Counseling"* w ) >: ' 

"Journal of Rehabilitation" 
'"Knowledge" . . ^ \ . ^ * . ^ /. • 

v "Lab Notes - Research, Utilization Laboratory" * ' 
^'Mainstream" " ; ; ' * - . 

: "Management World" • * ' 

; "Manpower Comments" . 4 « * * 

% ' "Medical Self-Care" . * • 

- ?|MentaI Health Reports", • ■, , ~J / >* f _ • 

- "TbeL New England Journal of Medicine" , ' • 
. "New : World for the Physically Handicapped" '* 1 . ' , * 

"N]^U^- Information .and 'Feature Service" • . V-'">,' 

"OccupationaL Outlook Quarterly" \ ' 



''Ohio Libraries Reach Out" 




6 



v £05 




sr./ »- . . * 



V." 



£C*"!j&^^^ Mer^cans , Report" 

' ;. '^^ur^ra* V.: ; ' ^ ' 

. '%ef P^rsb^ael ,and Guidai>ce Journal" 
"PathHnder" ; 
/ "personnel Journal" . % 

"The Pipeline" 
^Prevention", - . ^ . 
"Professional Report of the NRCA" 
^"Sharing" 

''Psychology Today* 1 
• "Rehabilitation Counseling Bulletin* 1 
" Rehabfiim Newsletter" • v* 
"Retiabiiitatlop Digest" 
• . "Rehab Indicfcors Update" 
k ^'Rehabilitation World" 
"Research" t • ' 
"The Resource" 
"SchizopKreni'a Bulletin" 
"Social Security Bulletin" 
. "Spotlight on Aging" 
"Statistical Reporter'.' 
"Stout VR Institute!' 
r "Training and Development Journal" 
"UC People, fete.'" 

"U.S, Journal of Drug .and Alcohol Dependence" 
"Word from Washington" 



# 



m : -: 



I 



gg, 



^9 
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m 

ft. 
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£RkiNIN& , ' BULLETIN 
(See Don for Details) „ ° * 



* Training . 

- MBO - . - ^ 

. Conference on- Know-, 
i^dge Use 



Locatjjpfl 

ft 

Ann Atbor, Mich*. 
Pittsburgh/, PA 



Preparing & Presenting ^ r 

.Oral* Reports kiddJretQwn, £A 

Evaluating* Health 
. Programs 

Evaluating Trng. 
• Erograms 



New Tork City 



Williamsburg 



Applied Statistics, for 
Decision Making * Hampton, VA 



Effective Written 
Presentations 

Program Eval 
^Review 

Program Budgeting 



Hampton, VA 



Albany, N.Y« 



. Ann Aj:bor ? Mich. 



Getting Results' froin * 

Surveys & Question- Anaheijn> Calif . 

naires , 

B. How to\effectiyely 

use formative & . • 

summative, feedback ' * 11 



. Making the most p£* 
printed material 

m -Selecting & using the 

, . right t;ask .and needs 
• analyses \ s 

\E. f Measuring & Evaluat— 
f y ,. irig -Training * 

Fv Needs/ Analysis: Process 
and Method*/.*; * 



^G. 'Writing for the Ear & Eye 
H. r ln Overview df Evaluation 



Tuition 
$525.00 

100.00 

130.00 

. 335.00 

286.00 

240.00 

240.00 

375!. 00 
^5 .00 



60.00 
Each 



February 10,* 1981 

*■ '* * 
• ■ -. 

*-*"*£" . 
Date's j' " " 

3/23-23y 6/15-17, 
-8/10-12, . 



3/18-20- 

m . 

3/4 & 11 

4/2-3 

3/4,5,6 

3/26-27 
4/8,9 ' 

5/5,6,7,8. ' 

4/6,7,8,9 
6/8, 9, 1Q., 11 

5/13,i4>15,16 



1 



4, 



.Penn. State^can ^hting^gfoiip .training to us in*,l. Statistics for Managers J ^ 
How? to. make effective Business & Technical? fre'^ntations, -and. 3 .^Manage-' / 
. -men^bj^ Objectiyes- * - % . *' ' "* . 



KSVPcSstt would be.. between $1,000 & $2^00 for a group Seminar % here. . ~ ^ 

-flerhaps as the/ Fiscal Year progresses and we %ill have extra money, we cpuld ^ 

W^:-£ko''£Ll : out- tor draining ., ' /:,..■ ' * ' . . :■' • 
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EVALUATION COMPONENT 
tLIENT ASSISTANCE PROJECT 



f 



- ♦ 




September 198Q 



* We* 



PROGRAM EVALUATION 



Program Evaluation is a systematic set of data collection and analysis activities 0 
Undertaken to determ^ne^the value of a program to aid management, program planning, 
staff training, public accountability and promotion.- 'Evaluation activities make . 
reasonable judgements possible about the efforts, effectiveness, adequacy, efficiency 
and comparative valu? of program options/ m • 

This program evaluation will involve the measuring of program objectives, development 
of a management information 4 system to collect information^and a dissemination and 
utilization plan. 

* • 

The mission of the Client Assistance Project is to assist cl ients in receiving all 
of the service's they are enti t led^r is feasible with available dollars,, see that 
they understand r their rehabilitation process as it relates, to ^them jmd^they are 
satisfied with'the. services" provided. • ' ^ * 

J. Program Objective - wi $6\ n \ the first- grant year assist 150 client/applicants 
to receive services they rare entitled to receive by entering the CAP process. 
They will enter the CARJPj^ocess frorn^ the^ fol lowing levels of client statuses: 

(1) Those placed in applicant status (02) 

(2) Two closure~"categor ies of castes closed 08 fcom Status 02 . k 

(3) Clients who expressed unresolved concerns in the annual .cl ient 
surveys conducted by the Program Evaluation Section . * 

(k) Those clients referred to CAP by external agencies or community 
' groups or BVR/OVB, • 



Program Objective Measurement - - 

A. Outcome Measurements - Number of referrals to JCAP, categories of 
assistance to cljents , « 

B. Process Measurements - Is referral method generating desired objectives? 

C. Timing - p Periodic evaluation on a quarterly. and yearly basis (see 
attachments) • • ■ 

D. Managementc^tffotmation System - Forms for collecting data (see attachments) 
reviews of data collected and consul tatiorv wi th (JAP and BVR/OVH /Staff * 

E. Utilization - This impact wfll be measured by the degree* to which the 
information is used. The following guides for di sseminat ion* wi 1 1 be 

• % used: 

(1) Awareness - information disseminated to the audiences \nd 
- information users • v ' ^ *' 

(2) Understanding - information should be clear and xonci se wi th 
opportunities t£ request .clarification 

(*3) Acceptance - valid 1 and reliable information, worthy of * • 
action and, . 
• (k) Internalization - timely information relevant to .tfye needs of 
the users 

2. Program Objective - Assist clients in understanding their rehabilitation 
grogram and determine if they are satisfied wfth the services . provided. . 

, Program Objective Measurement # < 

' A. ' Outcome Measurements - Number'cxf clients satisfied • . 
B. Process Meaurements - Reasons for satisfaction or lack of it, 

comparisons of BVR. counselor and/CAP client satisfaction, uniformity . 
- of individu§1 satisfact ion and reasons for lactof it. * / 

%T. Timing - Periodic evaluation on a -quarterly and yearly basis (see 
attachment) • > . 




./ Es > ; UUlizat'fon - This "impact wi'l I be measured h'y- the degree to which 

the information is used. The following guides '.for dissemination will 
Ce used: . . 

(f) Awareness - information-Tlisseminated to the audiences and 
information* users 

(2) Understanding - Information should be clear and concise with 
opportunities to request clarification 

(3) Acceptance - valid and relalble information, worthy of action 
and , 

Ik) Internalization - timely information relevant to the needs of 
the users' * 

■ tTC" 0b l eCtlve "^solve difficulties which clients maybe'having with 
.the VR system as quickly and amicably as possible. . 

Program Objective Measurement \ 



< 

A. 
B. 

C. 
D. 



E. 



Outcome- Measurements 
Process Measurement - 
difficulties. 



J 

- number and speed of positive resolutions .. 
degree of cl ient satisfaction with resolution 

Timing " Quarterly compilations and a yearly review 
Management Information System - Forms for collecting data '(see 
BW%VKs|af/ eVieWS ofcollected dat ! and- consultations with CAP- 
Utiltzatlon - this impact' will be measured by the' degree to which 
the .information is used. The following guides for dissemination 
will be used: 

Awareness - information^disseminated to the audiences and 
information users * - 

Understanding - information should - be clear -aod .concise with 
opportunities to request clarification 

Acceptance - valid and reliable information, worthy of action 
and , 

Internalization - timely information relevant to the needs of 
the users ' ' ' - 



of 



4 



0) 
(2) 

(3) 

C<0 



|rogca^0bjecti^. - Recommen-J modif ication/clar if icat i,on , if needed, of 
BVR^OVH Polices, and Procedures as related. to the client service del ivery 



Program Objective Measuremen t 



Outcome Measurements -.Annual review to be disseminated 
B. Timin ig - Quarterly compilations and a yearly review ' 
ManaoprrV>n tV In forma f inn Cvct-am » t>~*.t _ 1 1 j_ . 



Management^ Information System 
Jcpn s u 1 1 a t i on* dxmp 1 e t ed 



Review of alj data collected and 



D. 



lfe v ■ ' '' I "The<Progi;am-- : Ev 
Iff 5 :'." t&o&c t : : (CAP ) ' : 
J W¥-'r-' : -Progresses it " 



Uti^atiori - Completion of a foUow-up to.determfrie if recommendations 
in annual; review are implemented, considered or modified 

ition Section will work closely with the CImet Assistance 
KF , sense; cjiangtflg. need> for evaluation. As the CAP 

^ti.opated that additional process a'nd outcome measures wi# 
ti { llzed particularly in the second year. >£f* 




CAP PROGRAM EVALUATION FORMS/CHARTS 



jf Preliminary forms have been developed to assist in the .imple- 

|^,' - mentatron and conduct ot: the Client Assistance Project, Three 

are#s are of the most concern at this time to the Program Evalua- 
te'*** < tion Component': \ • . ' • . • 



1. Application * , . 

2, ^Counfeel6r Evaluation Form 

3 S Client Evaluation Fojrm t i 

It is anticipated that o'ther nflod^ will be developed to facilitate 
the*"prbgram evaluation process of the Client Assistance** Proi ect , 

Form 1 ^ Application This is* the Client . Assist ance„ worker 1 s 
intake tool and is used at CAP "closure, . * 

"> - . % - . . - ; . 

Form .2* - Counselor Evaluation - The BVR counselor will complete 
this torn and return t^BVR Central Office when a CAP* case he/she 
is involved in< is^eJosed. 




Forjn S--* Client Evaluation *- The CAP client will complete this 
form alter CAP closure a«Ad- return form to BVR Central Office. *^ 

Form 4 - Flow Chart at Referral '- Client CAP application^form 
flow chart . 

froym 5 - Flow Chart at Closure - Client CAP application form 
■ . flow crrart. \ ' n • " * « " 

V ' ' ; 

|^ Refinements to these forms are anticipated prior to project opera- 

Ik:, tion ..start-up. Several tables^and lists referred to on each 
Form. are noiv being developed. 



CLIENT ASSISTANCE 
CLIENT EVALUATION FORM 




firs 2 = s: ^ ent project ' we 

cbnfidential and will only be uLd fit reJearS pf«~! you provilt vill be completely . 
.in the enclosed Postage Paid Envelope ra " arc . h ' Please «««« questions and return the form 



1.' ^satisfied are you with the help prided by the Client Assistance Project?" 



_a. Y er Y Satisfied 
J3. Satisfied 



_c. Dissatisfied 

* 

Very £&ssatisfie4 



■• : r 



_a* Very Satisfied 
_b. Satisfied 



J3 . Dissatisfied 



ji. Very Dissatisfied 



' SienJ 2i^^;=^ Wi< ? ""J**" 1 ^^tation as a result of L help fron the 



Lient Assistance Project? 
ffis ( ) NO ( ) 



4 * SoSt ? feel Y0Ur Pr ° bl 7 W ° Uld ^ b — ived without the help of the 'client As^ce 



YES ( ) 



NO ( ) 



3. Are you indeed of further help from the client Assistance Project? 
*3SS ( ) . \ no ( j 1 



6. Additional Comments: 




272 



124 




. , ... _j . i 



CLIENT NAME 



NUMBER 



9 



/ COUNSEMR^ EVALUATION. OF 

w ^*& ® CLIENT ASSISTANCE " P,RO J£CT 
, ."-w. » » I - • - ■ ' * 

The above named client'has >een in contact ke Client .Assistance Project. The^following^ 
qifestions are being asjced *o help in the evaluation of the Client Assistance Project, felease • 
help iis by answering the questions and retaining the form in the enclosed Postage *aid Env elope. 

» you have any questions concerning the corfoletibn and/or **e of this form, feel free to 

telephone Harry W. Guise, Administrator. Prdaram EvAiii»i-<inn co^Hnn 



^ ^ answering tne questions and retiring the form in the enclosed Postage 

^- > ^ I* you have any questions concerning the completion and/or fee of this form, feel 
telephone Harry W. Giiise, Adlainistrator, Prdgram Evaluation. Section at 8-447-7312. 

h \ • ' :.?< „ • " * < ( " ^' - 

" \ . * to* 3 ** this client's problem or concern\have been' satisfactorily taken care of 

i »%o the .help of the- Client. Assistance^ProjeW? 




you'withou^ 



YES, 



NO' 



V 



2 ' -^ie|fe^ OU f6el ^ e 9f ient Assistance Project af f ecjfced v the rehabilitation' process of this 

BEu R , 'HINDERE D 
3,< rtla^o^hip?^ ^ Client fes f stance, project ser^ices)have on the cUent/VR counselor 



****** j > 



#•7-. 
"J \ 



- a i <a ( eat lX* improved client/VR counselor relationship. 

. Improved client/VR counselor relationship. ' ■ ^ 
_c. ^Had' kt^e or no affect on client/VR counselor relationship, 
jd. Hampered client/VR counselor relationship.'- 
_e. Greatly hammered client/VR counselor relationship, 



4. How satisfied ^^you'feel with the outcome wliich resulted from the Client Assistance 
Project's 'activities? ' f ^ i |£ ' . ^ 

* ^ 'Very Satisfied _c. Dissatisfied. * . ' * ^ 

^b.^ Satisfied -* ^ . , a . Very pissa^lsfied r _ 



5. Are you satisfied .with your understandin g o^Uw pu r pose of. the Client Assistance Project? 

- I*- Very Satisfied c. , Dissatisfied v ?• 

r _J J>- Satisfied d.: Very E*Lfatisfied u ' ( 



d.. Very E^jfatisfied . f 

6* Are you satisfied Vith the Client Assistance Coipselgi 
and suggestions when discussing tjiis client? m 




Very Satisfied 
Satisfied ' 0 



_c . Diss atis f ied 
^d. Very Dissatisfied 



7 . Additional' Comments ; 




.lijigness^bo listen to your ideas 



u 



rn i 





or. 



•i - 



PENNSYLVANIA CLIENT ASSISTANCE PROJECT 
APPLICATION FORM 
* PART I - 



NAME 



STREET ADDRESS 



* CITY. 



SOCIAL SECURITY.no. 
' DATE Of APPLICATION 



\ Month/Year 



STATE 



COUNSELOR 



PHONE " 

y 

CT^TUS* -OPE N 

REFEfckAL SOURCE; 
1 . C LIENT v 2 
5 



ZIP CODE 



Name/Nunbef . - 

OFFICE-PHILA. R05EHONT OVH (CIRCLE ONE) 



t CLOSED 



SEVERELY DISABLED 



YES, (2) ~ NO 



OTHER - SPECIFY 



BVR AGENCY $ 



OVHs^EWCY 4 OTHER CAP CLIENT 



50CIAL\AND DEMOGRAPHIt INFORMATION - 

DATE OF^IRTH^V (MM/YY) AGE 

S* 02 v(l»V7HITE, ,2=BLACK, 3«0THER - SPECIFY*" 

PISABIL^TIES AS REPORTED BY APPLICANT 



SEX____(1 ■ MALE, 2 ■ FEMALE) 

.) EDUCATION _( YEARS COMPLETE l5). 




mm? 



v » - . 

■ • : * ,-■ — ~. • ." v -^-i — - — ■ fe • 

-» , . , , < _ 



* "X- 





fitr 



7*^ 




J' 



'Application 
Ta Seen 



Weekly ypaa^* 
,'PorVtj For 
: trltrtcf - 



i ; > 



Applies utm« 



as 



"PilM At 



[ Upcjala To 



7 



C. /* -U 

CenTval* , OPvjce 



: Added 




-A 



3&f? .oc mftti on 
• .CAP £ 



r 




4 . ; ; * 





.AT CLOSURE 



CAP Completes 
/-•psii cation 
At Cfosur^ 



US* ' 
' 4 

- i\" v.:' - ■ ■ , - - 



Post On 
Weekly (/pdeuesg^l 
Farm For 



I 



0 . 



Aopiit 




\ 



'Aoolicatioa 

'frTcd At 
' CAP • " • 



\ 



Send iv'<*$kiy 
To Prsinct 



I 



rat 



£ Counselor 




t 

t 
t 
i 

t . 
t . 
t 

A" 
f 

i 
i 



EytilvafioQ 
\ Farrn 



Counselor 




i 



ft. 



EE 



■1; 




/i?eo.orvs ! 





It' 1 '-' 

; - 



It 



r 5^ 



\ 
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0VR-38O " \ ■ ** 

3*77 " Commonwealth of Pennsylvania - « 

* Department of Labor andtfidustr? 
BUREAU.pF VOCATIONAL REHABILITATION 
Harr isburg, Pa. 1 71 20 . 



FOLLdW-UP SURVEY 6f VOCATIONAL REHABILITATION CLIENTS \x 

5 , x • (CLOSED-REHABILITATED) . ' 

*° * urlr€cor ? s W se closedas being- rehabilitated. This means that you were suitably SmoloJed 
s , er Y ,ces from?lhe Burwu of Vocational Rehabilitation. In an effort to improve t^se^SZe7S 
.^e would like to know your current situation. We would appreciate it if you would answer the fwquSS^Sls 
23^^ !S V tUm U h * e self-addresSed envelope as soon as^riH^Sn S 

SS£3£t £ H 6SearC ^ ° n,y ', wi » ^ contacted. This study isESn "moS^ o 

improve services to handicapped people. Pfease help us by filling out.tKis form/ • \ . ,uu "*« « uy io 

* Phone where ybucan be reached: 




New Address: 



1. Art} you employed, self-employed, or a homemaker at this time? 
(1) D.Yes If yes, please give type of work 



r 



K, 



■t 



(2) □ No If no r please skip questions 2. 3. and 4. , 

♦ "® ; — ' 

2* ypu P'ease Indicate your current weekly earnings (before taxes)? $. 

& Are you doing the same kind ^, work j£pu were, doing six month's ago? 

.* * (1) O Yes 
; -f (2) □ No\ • f m 

% 4. How sa^tisOtd are you with your present Job? * - 

t(1) Very satisfied > - 

(2) Satisfied * \ . . # *' 

<3), □ Neither satisfied nor dissatisfied ..-./.'« 
(4)- □ Dissatisfied* 



(5) .□ Very dissatisfied 4 . *"";;;. ■ " ."• . 

i i * 5. How.many p^nths hav%you been- unempioyed since you have been rehabilitated? 

.(2)^;D' Lesstr«n.1.rnonth ' " • , \\ \* 
~. (3jv:0. t»3 months'" • * 

"f!"^' v Wt O 4< »tinths V . v * . 

--<5);rD 7^2mqn'His.^ ^'I 's " ^ - ; : * ^ : 

3^ ^tf)%P^Kfere than 12 months- U . . - 

- ' I. Do you need rehaiilitatioh services? ; (1) □ Yes 



1 1 . 

(2) □ No 











... /•/ ; \ ..... 










;;7-. ; / 



if 'v ; 



7 



BVR-Wt 

V77 : 



Commonwealth of Pennsylvania 
Department of Labor and Industry . * ' 
BUREAU OF VOCATIONAL REHABILITATION 
~ Harrisburg, Pa. 17120 



/ ; ... 



FOLLOW-UP SURVEY OF 
* VOCATIONAL REHABILITATION CLIENTS 

? ^ (CLOSED^fHABILfTATED & NOT REHABILITATED) X * ^ M 



6 6ur records indicate that you were referred to and may have received services from the Bureau of Vocational 
Rehabilitation. In an effort to improve the services we provide.' we would like to know your current situation. We 
would appreciate it if you would answer the few questioqsjjn this questionnaire and return it in the postage-paid 
self*addr.essed envelope as soon as possiblfe. All information is. confidential and is for research use only. .No 
/employers will tie contacted. This study is being conducted to try to improve services to handicapped people. 
Please help by filling out this form. 




Phone where you can be reached: 



4 • -5. 






New Address: " * 

* t 








* V 


- T 





'Check the one sentencertfiat best describes your employment states. 4 < , 

(1) □ I am employed full-time for pay. 

(2) □ I am employed part-time. I amseeking full-time employment. 

(3) * I am enjoyed part-time. I am not seeking full-timg emplcryment. 
C 44) □ 1 am a homemaker. . • • m 

(5) • -03*1 am attending school.- - . - ' . * r * * 0 

(6) , "tf \; art) not emplby«d. ? t am seeking employment; " 

(!) .0 I am not employed: l*am not seeking employment. - * dfc , t 

\ (8) . d NonSqfthe above.' " / W 

.>**«. » * . ' • . j w \ 

* 8esljde each of the Allowing statements check the line that best describes your feelings about the services 

•VOII r»Cm\vmti from IhA RttrASlt nf QAkaMK***!**** 1 



I* 



(a) My qounsef&'s^illinghess to listen to my ideas* 

V an&suggestions'fri my <:ase was 

-(b)' The Information that rp^cbunselof^pasyjded .in ^ 

regard to understanding my disability was . 

(c) . The length, of time, between my application an.d- 
, . )he tim.e I received- services y/as, v : V 

; *;^^;^.6 'plfcmptness with which ;your VR services 
. ; j; *wve "provided was i . . . .% ... ^ 

, / (e)-;;^e insults of medical services were . „,.,.'. . . 
* - • . , (ft-- ^he benefits from training I rec eived were . . \ 
C • ; m -%^!ntef^5t pi the people.1 "was sent to 



5Ai%^v^t>*" - ? th);>Oyeralj;r feet the heTplreceived was. . , : . 






' Ntlth«r Satisfactory 




SttUftctory 


Nor Unsatisfactory 


Unsatisfactory 


(D □ 


••* ,(2)0* 


- (3).D " 


<i)p v 


. . . . .. . .'(2) d . 9 :. 


0) □ : 


0) □ 


% . 


. ;(3) • • 










• . X2) 


(3)D.. . 


(i) □ -* 


(2) □ • - 


(3)GJ 






' (3) □ . 




"• .' _(2) a • 


(3) □; *, - 


> * / 


.• :■ .;(2)P : 


. (3)0 










% ' t 






4 , 






' '*xS^f%*W e * i9 whlch-of th« following services you received from the Bureau of Vocation*! Rehabilitation 
^^5p.)»nd-w.hjch ones yoi» found helpful t£. you. . 

• ■ *|>Hfte^iagnqstic and F 



c 



, vv ,. s ....- -EvaluatioD\(rpedical examination, 
• ^s^chojogicar testing) , . . !\ . ; 

(b) "%€cjtea| Services (hospitalization, surgery, ' 

o^rdatfrjent) : . 

(c) ArtlficlaT^PPliances (e.g., arm, leg, brace; ^heet- 
<:>taUfefc^^ ....... \ f.;.. 

(d) Vocational Training (e.g., college or trade school) . . . 

(e) Sheltered Workshop Training . w [ 

(f) On-the-job Training *..•!!!!!•*!!! 

. (g) Tools and Equipment \ \ 

(h) Help in obtaining a job ^ » 

(i) Pennsylvania Rehabilitation Cahter Training . . ..... 

Q) .-Dental T?t 

. (k) Glasses „ , . :../...;..[[ 

(I) Hearing Aid*. ; . . # \ / 

(m) Other . 

(specify)' • 



Tht< ttrvlce wat helpful 



□ 

□ 

□ 

□ 
□ 

□ 
□ 

□ ■ 
□ 

□ 



□ 

□ 

□ 
□ 
□ 

□ 
□ 
□ 

□ 
□ 



V ■ < 

4, Do you feel you were placed in a job that was suitable to your ability? * ' 
(1) □ Yes> 

. (2) □ No - 
^ <3) D Never employed after completion of vocational rehabilitation services. * - <* 

, 5e WJio decide^ the type of work for you? • 

(1) □ * I decided • . v ^ 

, (2) My counselor decided my type of work.. \ . * ' - . 

(3) □ My counsefcr and I decided together. - ' * • 

Please Indicate 4 bel 9 w how helpfuFyou feel your counslrlr was In.flndlng you embloyment or'heloina^ 

you to .continue your employment. - * ' 

(1) Very helpful/ - /(4) O Not helpful ~ tS * m * ] 

iov 2 J? elpfu ' Ul ,v (5) Q N)Bve> em P ,o y ed after completion of vocational rehabilitation services 
(3) u t Somewh,at helpful . — 

* 7v If you presently need vocational rehabilitation, whlcfr servlces below do you rerfulre? *' 

(1) □ Further Training ' ~ 

(2) □ A Job ; . • . . * % ; - ^ 

(3> D A Aeafing Aid ' ~ ' ' * - ... ^ m 

r . - • (4) Glasses / ■ > # . . 9 

(5) *• O A new artificial appliance (e.g., leg^rrrr, brace; wheelchair, etc.) 5* 
(?) □ . Dental Work . " # - ' - ' ^ 

- . ,(7) □ • Someone to talk to about training me^KaTlnding'me a job # * . 

. .(8) □ ptrler (specify) ■ ' ; - 

(9) □ Don't need vocational rehabilitation i services. . 

i . Would you like a Bureau of^Vocation ? l Rehabilitation counselor to contact you regarding further services? 

(1) □ A Yes' . *(2) V D No- .• ' - .* 

* • * . * • ' * • ». 

% K Woufd you recommend VR f services to <a drsabled friend? ' *- c/? • 

< . f.O) ° ' ,(?) □- No, -i ; ' - ; 

" t * \ • ji - ' " ' 7 s - ^ ' • \ ^ * ■ * ' * - ' * t 

1 ^ l^i™ i6i J 0Ur c^peratlo'tf In this stfdy: If you^hifVe any'comments ydu would like tb add, pfeaso> 
■ include them below: . ~ • •'. - a^^- ... , • ~*-* • —~. - c r . 5^. 




. A 



3 



•f ■ 



Appendix 20. 



%■ 

* * 



# , Cross lifeulation and Validity Chart ^ 

J "Cross-ta^ation'bjf Question *2h from Standard'9 questionnaire with Question 9 
from Standard 9. questionnaire (Validity) * 1 Question 9 

t * © . /yes m 

<Sa tis factory 
Neither .Satis factory nor .Unsatisfactory ■ 
.Unsatisfactory u 



Question 
^ 2h . 



XJ "Cross- tabulation of Question *8 from Standard 9 questionnaire with Question 7 
fron Standard 9 questionnaire. Of those Wanting further services f what type 
of further services do they want? ■ * 



Qu.ebtion 
7 



FurtheiM training 

a job , 

A Hearing Aid . ; * 

Glasses^ \ • 

A net/ artificial appliance 
Dental ^Work 9 > ' 

Sotteone^p tal/c to aijout; trailing me 

and ^finding me 'a job 
Other * 
£on 't ne§£ vocational rehabilitation 

services $ 



Question 8 

YES - ' NO 





































t 



If any field positiofi (448^456) has a numeric- 1 then do the cross -tabulXion 

III 'Cross-tabulation of Question 3 from Standard 9 questionnaire with R-300 services 
P^^dec^fields': Do the services receded by client reflect what is on the 
* ^ * f ~ 

V * Matches . No matches 




Question 

•3 



0", . 



% X - 



0 



Diagnostic and Evaluation ' ■ • 

fifedical Services * 

Artificial Appliances * * : 

Vocational Training 

Sheltered t Workshop Training 

On-$he*Joh Training 

Tools and Equipment 

Help in, obtaining. .a job 

Pennsylvania Rehab Centex Ttaining : 

Dental < • 

Glasses • " 

Hearing -Aid 

Other i 



d, 2, 3,4, 5; 



R-300 









* * 


* 




—a i 




















S 




^ P 


— a « — 






* • 









V 



• The -/oilowiny-cah^ersio^-tahl-e, should 2?e used' to/ <^fcure.Jn£orjnitionJ!fojtsCthe*» 
, above ;chaxt. a * . e c. " ■ 



'Diagnostic and* EvalhatiQn (fcl? or 420) with field position'26l ' . # v 
:\Jtedical> S^ryices^ (421 Wr 422) with field position '2%jt* 
^Artificial Appliances y (423 or 424) with- field position ' 
^°^^onaI^2!fM^iig C$25' or 4261 with field positions 263-266,268-269 
■ ' ^^^Mp^shpg Trai&Lrig (427 or 428) with f t i'eld A position 2$ 3-2 69 
* OnrW e . job Training >+C429 or 430) with field position 267^, ' 



-y r ' ?P^i s: * a&f Egulpmeiij: C4'3X 'or*432) tilth field' pogitioi) 271 



set 



it* 



f NOTE: 



- -.- . • - • v .-' ■ ■ 

' -Help in obtaining a job (433 ox 434) 'with fiel? position 271 . 
' . Pennsylvania Rehab C&ntet Training. £435 or 436) with field position^ 
1 263-269 < \ \'. : . /."?.-( * « 

Cental (437/ 6r "438) with fielti position 262 ' . . „ , „ 

^Glasses (439 oti 440) with field position 26'?* 
/r ' /Hearing Ai€ (441 or 442) with field position 262 ^ ' / 

' Other' (44 X^r 4 t 44) with field positions 261^271 * , 

£A has expanded tfcis service prpvided field to 'accept cddes 4 and 5. 
'Normally, RSA will only accept^ codes t O^thru 3* j 



Jlf,. Cross.- tabulation of Question 6 fzoti Standard 6 with. Question' 1 from Standard 6. 
' Are the previous clients who need further services workipg, not working or 

* -hpmemakers? * - • 

*\ '* Question 1 - * 

- * ~, Working Not- Working Xomemaker 

Question Yes ~ 



6 



No 



The following conversion table should be: used to capture information for the 



above chart. 



Yes (411) with- field position (402) coded .2 or® 3 ' 

Yes (411). .with field position (401) coded 2 

Yes (411) with field position (402) coded 1 

No (411) with field* position (402) coded 2 or 3 

No (411) with; field position* (401) coded 2 

No (411}. with field position, (402) -coded 1 . j 



Cross-tabuiation of Question 8 from Stan'dard 9 questionnaire with Questional 
from Standard 9 'questionnaire ; 

Question 1 . < 
- _J 2.3 J. 5 —6 7 - 8^ 

> .26 No ' • 



Question * 



8 



• „ ^ } 26 No 
'*Sr%Z8- & 4a. Yes 
28/ 6 'M No 

■4- 



j?iiestioft '8 [is in* field 9 position 4ST with^YES coded;!' and NO coded 2. 
'** \ Qu&stion l is in fierld position 410 with numeric codes of 1 thru 8. 

KT gomparison of earnings at closure from t&e R-30Q* with Questidn 2 from Standard 
6 .questionnaire* ~ K <' 




E^aOv^^^ , 5; i ^}xest^0r2 V 
' ?^ilow;--up 



gotal ifns. Average .rdtal ggns. Average 



iVbrJfe-Status f -earnings 'at closure 
Wor^S%atViS 2 : e$rnirigi at- closure 

WI?fC> "! 7 ? l "»^ : nn WbrJk St^t'tiA 3 : eattx£hgs at clokure- 

l^%f ' 1'*" >fctoyir / ^ti'tu^' ^'eaJMlitgi' at- closure 1 



JJseM'eld. position* 245 io^ select work status, and field, position 246-248 for weekly 
v-^-fc^-^f earnings^ ^ * - : - t: - "*~ 



'Never Employed , 
Somewhat »ot after co/npietio/j 



Question 8 



w 26 yes 
; 26 J\fc> 
2£«<?30 yes 
2£<S30 tfo 



, Somewhat *ot after completi 
Very Helpful Helpful gainful Helpful of VR sJZicts 
U ' i ' *r 



Question 8 is xn field' position 457 with vf<? i * „~ • 

Cuestio* * is to'iuu Zsition^Z 3g ^eSfcoL^o/? SS* . . 

^*^2f^<riSSS ^«»«*". ^*«ix.^ tt the series ' 

* • • * , ' _ Restoration 

Gues-tion ^ Satisfactory T : " ~ 

2h ^ Nehh& satisfactory Nor Unsatisfactory 
■ ^ ^satisfactory . * 



-festaraczo /3 rrai/u/ior a/id Training 



The following ; 
chart . 



coaverszi 



taMe-ahoiiId to used to capture irtforrca tip* for the atove 



ft" ' 



J* Cross- tabulation of Question 
• Standard 9 questionnaire. 



\- . Ver^h^Lpfui 
5 " ^Ji^pful • 

Que^eion 6 .? omewh at helpful' ' > 
•. ' tfot helpful 

Never 'epploySd^after - ; 
powpie^ion "of r^i? services" 



\' .Question I / 











' t 
















V 


























Aj 


















9 


- «i * v 


, w 




« 





0 4 tMt /o«tl<» Wp.xiO.' iureric cides i u,r* 



^5 



£ ' . XZ Cross-tabulation of Closure date (MM/YY) on the R-300 with Question 5 
* Standard 6 questionnaire (Validity) ♦ 

f 1 . . 

(:\ r Le$s than * 1-3 4-6 7-12 More than 
£ \ " ^None 1> month- Months MoMhs Months 12 months 


, , Closure date 16/75 • 
' Closure date 11/78 
fl< " Closure date 12/7'S 
\l * Closure date 01/79 
p • Closure date 02/79 
IV ' ^ Closure date 0J/79 
£\- K-300 Closure date 04/79 
Iu.a._L*._- - i * ' Closure d'ate. r 05/79 
l\ Closure date : /06/79 
\ % ' Closure date 07/79 
f : ' Closure date 08/79 
, .» Closure date 09/79 






, , r 
















* • 














7 ' —W • 

\ 




















































— 1 — ^ ■ — 








1 










































7 








f 1 * \ • 

* Closure dpte is in field positions 273-276* 

\ ^%*Question 5 is in field position 410.^ 



*II ( Cross*- tabulation of Question '3h on Standard 9.^questionnaire with Question 6 on 
Stahdaz$k9, questionnaire , (placement* factor) 

^< - . . , . Question ~6 - . 



Question 26 Help in obtaining a job 
■ 28B30 Help in obtaining a job 



Never Employ. 
Kery " > Somewhat Not after comple* 

Helpful Helpful Helpful Helpful of VR serv. 



3h 



i 



Question 3h is in field positions 433 or 434 coded- 1. 

Quhstion 6 is in field position 447' with numeric codes 1 t£ru 5* 



>> -*& X %M Cross-tabulation of Queatiqp-* 3h on Standard 9 questionnaire with Question 7(7) on 
Standard J? questionnaire, (placement factor) * • 



Question 



' • Someone to ' tal/c to about 

, | : „ \ training me and f hiding me a t jbb . 

26 Help in obtaining job . ! 



^ --v — — " ~ * .« | — 

|>A* > * ^ V % 2&&30 Helj> in .obtaining ayjob- | > 



1st- 



Question 3h is in field positions. 433 6fr 434 coded* 1. , ^ V 

*<Ques€i % oh'7 (7) is in fielS* position [454 'codecrl. * * ' " 

V \ " ' \- . * ' • • 

XIV £evrelopme/jt o£ a profile of Question 1 (6) s on Standard 9 questionnaire with' 

Question* 2(7) on ^Standard 9 questionnaire. r t ^ * * 

/ Example,: Not employed-:seeking employment VS. Not emplotjed-not* seeking employment 



Average *age ' 

Sex* . • * . . ' * . 

status at referral 



#?A^v».- >*rtorA*. status at i;efe 
f^&%?<r . "Ty p&&fffilsepyi~ties, provided/ • ' 



0^ *XsSDf/ t SSI/PA at, closure 'type 



y 

18-29, 30-39, 40-49,^ 50-64 
MALE-] . - ; . \FEMAL& '' * 

+l*2,3 f 4,5,6,7,8, or 9' 

3 digit codes (fine-breakdawii) 
,'DSE, P.Ri, 'tRNG,"ETC._'_; % : 
.Receiving v ' * w . 
digit codas 




1,2,3,4,5 

11 less, 12, 13-16, 17 or more 
400 higher (Sum of Fed) 



•Marital statu& . , » 

* Educatioh * 
: Severely disabled ; 

XV Cross-tabulation of Question 1(6) on Standard 9 * questionnaire with Question 7(7) 
on Standard 9 questionnaire ^ Question* 7 (7 ') 

* * . * ^Someone to tali: to aJbout 

training- roe . and finding me a jojb 

Question 26 not employ/seeking employment 
* *Z(6) 28&30 not employ /seeking employment 



L 



. ^Question, 1(6) is in field position 410 coded 6. 

.. Question 7(7) is injield position 454 coded i. , < 6 

•-YVT Cross-tabulation vf client age on R-300 with Question 1 on Standard 9 -questionnaire. 

Q ^ v Question 1 ' 

• ■• -1 2 



3. 



8 



Date 

■ Of 



,15-29 ^ 

-^bJs 



Birth- 



40-49 
50-59 
60-64 
65s over' 







































4 


















■A 
















/ 



























Birth Date is in field positions, 79-82. . . — ^ 1 

. Question 1 is in field position 410 with numeric codes 1 thru 8. 

XVXI Correlation of the AH Service Cost Total joxl the R-300_ with breakdowns at 
hundred dollar intervals with Question 2h on Standard 9 questionnaire. 

- - .< Question f(h) > / v . • , ■ % 

. .* , -Neither Satisfactory* 



• Cosfr 



0-100 
100-199 
.200-299 
300^399 • 
400-499 
etc. 















• 




















• H ■ >■ T ■ 





SE£^w?7 --^e§i^c)A* '2h4ar; in field position 410 coded i, 2,^r 3*V . " 

^ ^ * * YVTTT r»Arr 0 7 -,f^« =« = 7,.^V^ • i 7 " W^&f-.' v _ , V , T 



W^'C xyf*?i Correlation analysis of the All Service Cost Total on the R-30Q with breakdowns 
^ a £ hundred dollar intervals .with Question 4 onStmdh.td 6 questionnaire. - 



. Very 



Neither satis fxed.' 



Very, 



\Cosk 



Wj&ip "'••« "-'^ 200-29.9 



Satisfied Satisfied nor dissatisfied Dissatisfied Dissatisfied 




i *• * 








C ^ ' 








L 4 * J 













































Sield position 176^178 packed*. 
.409 fodediXtJv:u5.., 



fc.' 



Cross~tai?u2ation of Question 2 (f) on Standard 9 questionnaire with Question 3(d) , 
• * 3(ef, 3J£), and. 3(i) on Standard 9 questionnaire (validity). * "~ 

_ - * Sheltered 

„y Vocational Workshop On-the-job 

Training Training Training 



QuestlofTZ 

The benefits 
from, training 
rep&zved 



Satisfactory 
Neither Satisfactory 
- nor Unsatisfactory 
Unsatisfactory 



PRC 
Training 



• Question 2(f)^ is. in field position 4l6 coded 2-3.' 
v ^ Question*3'(d) is in field positions 425- or 426 coded 1, 
Ques±ion 3(e) is in field positions 427 or 428 coded 1. 
Question 3(f) is in field positions 429 or 430 coded 2, 
' Que*$tion 3(i) -is ih' .field' positions 435 or 436 coded 1. 



f 



XX Cross-tabulation of Question 2(b) .on Standard 9 uy questionnaire with the' Ma jcpjS 
Disabling Conditions on the R-300-. . - " *'j * - 



.£>ues €ion2 (b) 
Information 
provided by % 
counselor in 
•helping cli$qt~.' 
understand • 
their* 0 disability 



R-3Q0 Disability Codes 

mo '-no, 120 130 ' uo 200 ''210 



220 - 



Sa tis factory dm% 
Nei ther- Satisfactory 
nor Unsatisfactory 








• • 0' 




















* 








Unsatisfactory ; ^ 








v* 













' Ques€xon 2(b) is in t fi)sld- position 412 <:gded 1-3. • ■ ' - " 

'Major Disabling ^Condition is in P field positions 138-140. mi 

XXJ <?foss-ta2>y2ationof Major Disability todes^HQ., 510, 522 oh the R-30Q with 
Question 2 (a) on' Standard 9 questionnaire." 
_ . - Question 2(a$> *• 1 

: . ■ Neither Saiiisf&ctQry f 

Satisfactory 'Nqr' Unsatisfactory 

500* 



c 



R-300 
^-Disability 510 
' 522 



' Unsa tis factory 



'£*>■■' 



-Major Disabling CpndPtion is in field positions 13S-140. \. * 
QppS?tion 2Xa) is in field positional! coded 2-3: \ 

tf#TJ ' Cross-tajbu2atip/l of £ue%.tion 2(g; on Standard 9 questionnaire with Question 3 
^on Standard 9 questionnaire. *" ' * . ' 



;?{ *>;vC> . , * > — — 



Question^ 2 (g) « • 
The interest .of people Satisfactory ' 
the- client, was sent "to Neither Satisfactory 
for sexyides ■ * 4 ^ Nor' Unsatisfactory 
, ; .. • ' ' Unsatisfactory M 



f. 



in 



Question 2 (g)\is in' fieid position, 417 coded 1-3.* 

Question 3 is in field positions 419-444 coded 1 as foilows; 

' 3(a) Diagnostic and Evaluation (419 or 420) 
3(b) M$dical\Services (421 or 422) 
3(c) Artificial Appliances (42^ or 424) ' 
3(d) Vocational Training '(425 or 426) 
3(e) Sheltered Workshop* Training (427 or-428) 
1 3(f) 6n~tJie-Job Training (429 or, 430) 
3(g) Tools and Equipment (43X or 432) 
3(h) Help in obtaining a job (433 or 434) , 4 
3(i) 'Pennsylvania Rehab Center Training (435 or 436) 
3(j) Dental '(437 or 438) * 
3 (k) -Glasses (439 or 440) 

.Hoaxing Aid l^l^r_Jt42J_ „ : 

*3(m) Other (4r43 o$>444) ^ 




COMM0HKMT^OFP£NN$yiVANfA^ % 

v . . - - - 

8VRO80 



APPENDIX 21 N 

. DEPARTMENT OF LABOR ANO INDUSTRY ' 



NMMJ OfWaTIONAL REHABILITATION 

HARR^BURG. PA. 17120 
\ 







> 










c 



Phone where youicon bleached: 



New Address; 



7 



know your current siSion Z/l Z . S ^ P 9 SemCSS proWde - we wouid iiie <° 
this qL*^ 11 Te ^nT&SSE^ 0n 
■in/ormotion is con/identiaJ SesearcTusaSv M " S °° n " P ° Ssibie - AU 
This study is" being conducted to Zr2 ^WPloyers will be contacted. 

□ D. I flfltn a TA7orr» j » . * * 



-h 




0 C^TSSST 8 *^"**» -nrple. a place like a workshop for 'the blind or 

□ c. I am self-employed. 

□ g. I am ,no( ^rkmg at present. I am: fPleose check onej ' 
, v □ v A student. 1 

& 2. Retired. . • ' • * ■ 

□ 3. \ A trainee. A , 

□ 4. r Qtherffleqse explain y ^ 



Q h. My situation is, not listed above (Please explain) 



life ^ ^ 



3 " ~ U C " IC!iB> ° r * ' N ^«.ON PUASI SKIP QUESTIONS J „„ d ,' ^ 

□ b. a £am not working 

□ C ',1 don't know. 



• - ; Ob. lml^^r^j£^ m ° ath ' * **** 



> □. c; * I am notfWorking. 
* D..4 Idoa't lenow. 



! 




C ^ WJ«H. your Urgert wurce of iupport? (P/eose chec t on 8 .) 
H-t. Current earnings from a job.' : 

□a F^^^^ M ^^; •••• 

•wt). . 5 wWch ™ of * 6foUo ^typ^ 

. '• Dtr Supplemental Security Income (SSI) for the Blind. - 

• nZ Supplemental Security Income (SSI) for the Aged. < 

• □ I Sf? SeCU T^ 10001,18 (SSI) for °» 

' Sit ^aSrS abUity Wanca \ " 

D . }• Private Insurance. * ' • ^ 

□ L Unemployment Compensation* " * ' 

□ i Other (Please exp/omj 

'□JjL'IiKMt l„i mmtt . •• 

,u o. None. * * « » y 

, □ c. I don't remember. * * 



ft 



&5" 



& • 

fey- 



. ^ .^Satisfied • □ 2. Satisfied 

- ^ *• Vory Satisfied □ 2. Satisfied * U 3 Omm^a ' ' 

U 3 * i^U'fied □ 4. Very Dissatisfied 



you? 

□ 4. Very Dissatisfied 



- «. m u D. IdotfUemember • q c. No • * - * 

Ify^how^tisfi^ > 

Very Satisfied. »Q -2. Satisfied □ '3, Dfuatisftod 

¥ u I. don (remember * Q c- No * \ 



□ 4. Very Dissatisfied* 



If ift, how satisfied we're you wp this help? 
□ 1. Vary Satisfied □ 2/Satfafled 



^□•3. Dissatisfied . □ 4. Very ^satisfied 



288': 



110, 

I 



v " ■ ' " 




10. 



Did your counselor arrange for yo'u.to have TRAINING? (PJease check onej 
□ a. Yes □ b. I don't Temember D.c. Nj 1 

If yes, how satisfied were you with the kind of Jraining' pou received? (Pieose^check^ 
□ 1. Very Satisfied 2Ts«tl»flad V □ 3. Dissatisfied^ ] 

How' USEFUL were the>efvicM^you received from the rehabilitation program IN HELPINi 

situation? (Please ptieck one.J 0 !* " * < y 

I □ a. The services were of great use. * • ' % * v " % 

— □ b. The services were of some use. 1 

□ c. The services were of no use at all. , 

□ d I receivdd no services from the rehabilitation program. 




At Very Dissatisfied 



TCCG^ybur present ' 



\ 



11. 



How USEFUL were the services you received from the rehabiUtation program IN HELPING YOlJ^O P&FORM in your present, 
situation (even if they were of little or no use in helping you get it)? (Please cjieck one.) 

□ a. The services were of great use. . . 

□ b. The services were of some use. ^ 

□ c. The services were of no use at all. r 

□ e. I received no services from the rehabilitation program. 



/ 



12. 



13. 



may allow other family 



it. 



15. 



Sometimes, after receiving rehabilitation services, people are better abfcftb care for themselves, 
members to work or work longer hours et a job they already have. Is this to in your case?(PJeasb check one.J 

□ a. Yes □ b. I don't remember D a No V 

- ' s • ' - V 

Sometimes, as a result of contact with the rehabilitation program, people find their personal attitudes change t How do you 
feel about YOURSELF now, compared to before you were in the rehabilitation program? (Please effeck one.) 

□ a. I feefthe same about myself. f ^ 

□ b. I feel better about myself. 

□ c. I feel worse about myself. 6 

How do you-feel about VOUR FUTURE* compared* to before you were in the rehabilitation program?\(Please check ane.J 

□ a. I feel the same about my future. N e « 

□ b. I feel better about my future. • > \ 

□ c. I feel worse about my future. f / 1 • sN % 

Has your ABILnYTO DO any of the following functions of SELF-CARE changed since coming to the rehLbilitation program? 

(Please check on tjie right side whether your ability to perform each item has stayed the same, improved, fr gotten worse since^ 
coming to the rehpbiiitation program.) ' \ * • 

x } \ Has stayed /Has Has got|en\ • 

/ ' the tame improved worse /. 



w - 

J~T * • 

mix- > 
- ■ ■■. 

SE&I£ 



a. 

b. 
c. 
d 
t. 

X 
v 

h; 
v U 
I 
L 
L 
m* 

XL 



Eating ..... 

Bathroom functions 

Taking meMicine 

Grooming {shoving, hair care, make-up. e'tc.) 

Dressing end undressing* 1 

Moving my body weight *. . 

* Moving around the home *"? A 

Climbing stairs - *< 

Heading . .'. . .' '••jj t 

Writing •«•••• • • «ftJ 

Talking ;y • ;« * ## 

Answering the phone * - 

Moving on the street /. . . .*ff * (1) 

Using public transportation .. . J . . . . (1) 

Driving a car * in 



Other (specify). 



a 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 

(l)t) 
(1) □ 
□ 
□ 
□ 
□ 



(1) 
(1) 
(1) 

W 
w 
w 
w 
fl) 



(2) 
(2) 
>(2) 
*2) 
(2) 
(2) 
(2) 
(2) 
(2) 
(2) 



□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 

a 



t2^a 

(2) □ 
(2) □ 
(2} □ 
(2) □ 

.(2) a 



P) 
(3) 
(3) 
(3) 
(3) 
(3) 
(3) 
(3) 
(3) 
(3) 
(3) 

;(3) 

(3) 
(3) 
(3) 
(3) 



□ 
□ 
□ 
□ 

□ 

□ 

□ ' 

D 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



s 



i£h§*:\ ; .-'4't^: •••• * 



; 1« 



■t » ■ . • 



if 



w: :.- • . . • ■ ■ • -s • •. . • *\ 

lb <. - •' • • 

|£v v; 16. If you mostly work as a HOMEMAKER, has your ability to do any of the following skills and Activities related jo 

HOMEMAKING changed since coming to the rehabUitation program? (Please check on the right side whether your ability to 
per/orm each item has Stayed the same, improved, or gotten worse since coming to the rehabilitation program.) 



(i) a* 


(2) o 


(i) □ 


(2) D 


(i) a 


(2) O 


(1) D 


(21 O 


(i) □ 


f2) □ 


(i) o 


,(2JO 


(i) o 


(2) O 


(i) o 


(2) O 


W ci 


-(2)'0 


(i) □ 


(2) O 


U) o 


(2) O 




. .. —TO BE ANSWERED BY HOMEMAKfeRS ONLY „ ■ . Has stayed Has * 

t ^ - „ the same -improved 

* ' • a. Caring for children and family . 

. b. Caring for pets. • m 

' c. Cooking and fixing meals * . 

d Washing di9he9 » 

e. Qeaning house . . : 

* # f. Doing laundry " 

g. Shopping for food and groceries 

. * h. Running errands . .*% » * 

, . i. Budgeting and paying bills 

. ' J. Taking pert in neighborhood activities . ; . . . . . .vr. . . . 

. ' k. <Jther(speci/yJL_; 1 • 

17. Have you improved in any JOBtt&LATED SKILLS ANli ACTIVrnES (like typing, using tools and machinery, assembly work, 
serving people, and so on) since coming to the rehabilitation program? (Please check one.j 

« *Q a. Yes . 4 □ b. No - 

Y * . * 

18. Would you recommend vocational rehabilitation services to friend? [Please check one.J 

* . , x □ a. Yea d b. No ? _ X 'i 

19. Would you like a Bureau of Vocational Rehabilitation counselor to contact yflru regarding further servicesrfPlease check one.J 

□ a. Yes .□ Ntf * 

* ^ H yes, Vrhich services belw a^yma require? - >| * * 

# . *' * C]**a. ^Further training ' - tl *. ♦ 

* K . Ob. Placement assistance is finding me a job f . * 
\ „ r t O a . A hearing aid 

•* " CJ*& \ Glasses'" . ■ 

. □ e.. A new artificial appliance (e.g, leg, arm, brace, wheelchair: etc*!) 

/ V ■ □ f. Dental Wocjfl * 

K f ~~ * □ g. Someone to talk to about training me and finding me a' job 

, □ h. v Other (speci/y] 1 '. , ■ 



* 20. Who answered this questionnaire? (Please check one.) 

* * O a. I anaweredthis questionnaire oy myself, 
"* □ b. . I answered this questionnaire wjth the help of another person. 

* *V ' '^SP* . O c Another person^ answered this questionnaire forme. ^ V 
* * ^ * 

- ( ' 21. Thank you for your cooperation in this^urvey. If vou have any comments you would .ttke to add, please include ihem hew: 









1 ( 




e 
















«. 


* * 




% 




















1 












1 1 












' ' ' '-. X 


•t « 
. % 
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' • ' t . APPENDIX 22 

• v ' • . * * 

f SIMILAR BENEFITS * 

• The data from the fitst p^ase is being collected and analyzed." A 

formalized structure now exists and is operating successfully. What. follows 

/ ' ^ 
,is % chronological listing ybf the steps that resulted in the formalized 

structure which will, it i/S believed, contribute enormously toward the 

realization of the goal ojf optimum use. of* similar benefits. * , 

1. Through the case' service review the need for t^e development of a 

specific sin«[4p*r benefits procedure was recognized'. Similar bene- 

fits, it was apparent, were* not being ^utilized Uniformly throughout 

' f . . IT','.- 

the state. "This was due, it was thought, .at least in pert, to the * 

fact that the BVR Manual bf Procedures did not' deal, with similar 

benefits as a -specific subject but'rather as % collateral issue in 

other sub jeot^ areas. * * ° 

\ 2. As much input 'as possible. was obtained from representatives of the 

Rehabilitation Servi9es ^dtoinis^tratibh (RSA)*, the University of 

-Denyer Regic^ffel Rehabilitation* Se^earch Institute (RRR-I) % and the 

Policy and Procedures and Field Office Sections of Pennsylvania BVR ■ 

on^such things as a definition bf<» similar benefits, sources of t 

^ « ** * • * * * • * 

similar benefits, .possible- procedures for documentation oIl similar . 
"v* / " * \ "\ - X •* ' 

benefits usage, data collection method^ , etc. ' • " 

3. The^information obtained througti the efforts-'in 2 above was- compiled, 
studied, and sent to the Policy 'and Procedures and Field. Office 
Sections for review. • • t\i * v " * », ■ 

4. The Evaluation. Section, met with r^jiresentatives of. Field Operations 

■ ' . • * . * ' 

and Policy and Procedures Sectionsrttto discuss, alL» aspect^ 'dissimilar 

^ , . # . m \\\ 9 a 

K A . benefits, including tl^e plan for collection of data via tele- ™ 

* , * ^ * • * ** ** , 



* 



6' * 



. • processing units* %\, v # * ; — . 

fee-'-.-. • • ■ - iv. • % .» » - • » 

■Kg"- . 5. The Evaluation Section met with representatives of , Field Operations . 



and Policy and Procedures Sections* to discusi all aspects of simi- 
lar benefits*,- ^including the piah for .the collection of d6tk via 

teleprocessing units ♦ - 

r » 

6* .Evaluation, Policy*and Procedures, and Field Operations Sections __ g 
met' with the Assistant Director of the ageficy to. discuss A) ttfe 



emphasis that should be placed on similar benefits, B) the ja^Jbifod , 
devised for collecting and reporting data on similar benefits, 

* - C) the development of a chapter in the Manual 'of Procedures devoted ^ 

- V • **. • V . ^ . . K ^ 

' 5. . * . 

exclusively to similar- benefits, which woul4 include -a directive*. 

' . that eae^ "district office was to develop and maintain a directory 

•* - > ' 

of similar, benefits^ for its. .district, and'D) feedback to the Uni-' 

v . . . 

versity of Denver "RRRI. This meeting resulted. in .a decision to 

A * 

release simultaneously in July or August 1979 a manual chapter on 

^ . * : '/ • ) • * * - — . *.? r 

^ *- similar benefits and a njemorandum on the plans, .to collect similar 

benefits data, and a further decision, in t 'f avor of a simplified plan 
of collecting and reporting the use of i similar benefits to be imple- 

* • , 

• mented October 1, 1979,* which wpuldjp attempt tV use existing pro- 

- " cedures in so far as possible to collect the data in order to avoid- 

unnecessary conftision and difficulties, B) cfaver closures 08 frocq 06, 
0 • • * . I • : * * . 

* . 26, 28, 30, C) utilise the teleprocessing capabilities of each dis- 

t'rict office, and D) for fcije present Jimit the data collected to 

^identifying tho^e casfes that used similar benefits and the categbry 

* * • • 

of services ' in ^[hich .the£ were utilized, using the following coctes:- 

C 4 < . * • • • > • 

0 - Servicfes. T • « » 



1 - With 3VR costs only 



4- ■ 

2*- Without BVR costs (clienf participation) * m *• 

3 - With and without BVR* costs (client participation only) 

4 - Without BVR costs (other agencies and" organizations) 



5 - With and without BVR costs (other agencies and organiza- 

• 

, tions 6nly) 

* e < 

.KCodes 4 and 5 indicate* similar benefits utilization.)' 

It' was also agreed t^at expansion of the collection of similar* 

* " •* ■ " * ' ' ' 

benefits data„would occur as the need arises. ' * ' " 

7. 'The Administrator <of Evaluation," wfyo is the project manager for the 

MEU contract,- presented ^e tentative^, plans to the regional and 

district administrators at their -bi-annual- meeting in Jjine 1979, 

eliciting their comments and suggestions. * 

- t * • 

8-. Following the above meeting Field Operations , Policy and Procedures,, 
* and Evaluation Sectionsfme't tQ outline, a manual chapter on similar 
benefits, which chapter was subsequently drafted by 4 the Policy and 

' ; . — f 

* Procedures Section, I 

M . 
m >. Prior to th£ ^o^pletion of the finai version- of the chapter and j.ts 



1 release to/field personnel for 'inclusion in thejir manuals the draft - 

> . •"■ • - ' « • S?'< - 

'f . of the chapter was circulated among selected field and other con- 

•'r . * denied personnel for commefits and suggestions. • * ' 4 ^ 



10. FolloVing. the distribution of the manual chapter on similar'benef its 
and the memorandum pn data* collection for similar benefits to fi§ld 
personnel the' Evaluation Section received numerous telephone calls 
-from field personnel' voicing trie need for a more specific definition 
1 W similar benefits that? -would reVate to the format used the re- 
porting process The number of fespoiises^ necessitated a meeting of 

M" { - • • • > * , * # - ' ' ' . s * ' * - ' 

M / * the Evaluation, Policy arid Procefllirei, and Field Operations Sections 

' : • ^ ■ * ' 

7' «' . to clarify and modify definitions and procedures . 

U, <The resulting clarification and modification ^or the chapter in ttfe 



If 



V 



. Manual of Procedures Was .submitted to a representative, sampling of • 

V'/" district office, fields personnel and^theii distributed to all field * ; 

iteiv" •• ' " p era ° nn ^- ' * ' ^ n 



•/;'* • •••• .• - ,\ ' -'■ - v 



12. On October 1, 1979 the new similar benefits procedures went into- 
s effect and data collection oh closed', cases began. * ' " 

. 13.^ Computerized quarterly reports showing. similar benefits usage are 
- / prepared by the- Evaluation Section and distributed to the district > 

' ^ , offices. * * , # 

The first phase of the task has been -successfully completed - there is" 

now/a formalized structure* that can identify clients that used similar bene- 

■ fits and identify services in which similar, benef its-^ere used. The second 

phase is the expansion of data collection. On April 1, 1981 a pilot study 

was initiated in one of the district offices for the Collection of addi- 
... # 

tional data. * ' • - . 

*' ' ' 

On three different occasions prior to the_ start-up 'of the pilot project 

there- were input meetings with the counselors', supervisors, AssiTta^t. Dis- 
trict Administrator, and District Administrator, in the district offiL. The 

**♦ *• . v • •- • 

purp'6s.e was to develop workable procedures for counselor docui^ntaiton of *' 
new data to; be recorded, and' to determine possible impact on staff and • 

• clients of. the new' procedures . The guidelines used 5 in 4 developing .the prol 
cedures were the same aS those used in developing the original program. 

1. Start wit* basics-, keep, it;, simple . " r e 

>. t y : m • j , \ • -4 * * . 

^2. '"Utilize existing procedures whenever pos^Jble. ' ■ '» - 

3. St^rf with uniform procedures/ * ' * 

»■ * 

4. Modify, clarify, or change a procedure whenever appropriate. 

* ' \ - A * te F the P r °cedures were agreed upon the entire "district staff -was- / * 
trained - managerial, professional, and clerical. — s 

■ . « 

■ The new procedures will provide information on similar b&nefitS usage 

in both open and'closed cases as follows:^ * * , ' 

.' * * • . ■ % y 

" 1. Identify, client cas^fS* in' which similar benefits are used* 

2.-.*, "Identify the sources of Similar benef its. 'utilized. ' " 



3&T , 

f^' B 3. Identify the services in which similar benefits are used. 

R ' . * ■ * 

fcf' * 4.* Identify cost savings resulting from similar^ benefits usage* 

- * • . ' ♦ ' 

- .Four computerized reports have been. developed -to monitor the activi- 

i\ : ~ ties of the pilot project. When the pilot project is completed - assuming 

>. • / _ 

!'* " successful outcome - the procedures <will be implemented statewide and 

^ Pennsylvania BVR will have* a valuable source bf'information for the support 

' ** 

y> »pf decisionmaking for t^e optimum utilization of similar benefits. 

: :"• . * Appendix 15 shows application of similar benefits utilization to 

* ' - ' \ 

* ¥ # ^rehabilitation program principals. 
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Govgrnor'ProcIaims 
IYDP in Pennsylvania 

Governor Dick Thornburgh issued 
jt-proclamapon chat officially added 
-Pennsylvania to the ranks of com 
-xnunhies celebrating 1981 as , the 
International Year of Disabled Per- 
sons in a personal appearance at a 
state-sponsored Nvor^hop for IYDP 
"community partners", held Feb- 
ruary 25 at Camp Mill. The governor 
vas one' of four governors chosen 
last year to serve on the board of 
directors of the U.S. Council for 
jfe>P t the national organization 
coordinating the year-long United 
.Nations project. 



George C. Lowe, Jr. Named 1 
Director of Pennsylvania BVR 



A nationally known human ser* 
vices administrator, planner and 
educator has been selected to, head 
the Bureau of Vocational Rehabili- 
tation. * 

The^appointment of George C. 
Lowe, jr., a former director of the 
Governor's Planning Commission fpr 
Vocational Rehabilitation, was an- 
nounced by Labor and Industry Sec- 
retary Charles J. Lieberth, whose 
department oversees BVR's rehabili- 
tation, training and -employment 



Also attending the \vorkshop were ? services to . the Commonwealth's 

Ginny Thornburgh,* the governor's physically, mentally and emotionally 

wife? Secretary of Laoor and Indus- handicapped^ , % 

((continued on page 3) "Lowe's selection ended a six-month 




Gov. Dick Thorn burgh displays certificate presented him. by Alan Reich, 
president of the U.Sl Council for the* International Year of Disabled persons, 
m recognition of Pennsylvania*! designation as a "flagship state" in the year-long 
National observance of IYDP. * % 

i, 



nationwide recruitment effort by a 
five-member, independenf search^ 
committee to fill the BVR director- 

"With his extensile background 
0 in the delivery of human services* 
especially to the handicapped,- 
George Lowe will be a welcome 
addition to the Department's 
management team," said Lieberth. 

"It is especially, gratifying to 
secure an administrator of his cajibre 
during Pennsylvania's observance 
of the International Year of Disabled 
Persons, which has been endorsed 
by Governor Dick Thornburgh, who 
is. a 'director of the JJ.S. Council 
of IYDP,'* said Lieberth. 
^ "His experience, will prove invalu- 
able^ as we implement Governor 
Thornburgh's policy of holding 
dpwn administrative costs so that we 
can maximize the direct services 
offered to t)ie Commonwealth's 
handicapped community, especially 
to those who are severely disabled." 
added Lieberth. 

Planned MH/MR 
Lowe's ' state government exper- 
ience from 1964 through 1969 in- 
duced the position of director of 
planning for the Department of 
Public Welfare's Office of Mental 
Health, which laid the groundwork 
for the Mental Health and Mental 
Retardation (MH/MR) Act of 1966. 
The legislation resulted in the 

(continued on page $) 
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Client, Counse 
Surmount CP 

Tom Koons, York D.O. 
Community Relations Rep. 

Client John Foschia of 528 East 
Washington Street, Chamtfershurg, 
whose mobility is lirmted by cere 
bral palsy,^ is the kind of person 
who likes to ^battle the odd§. "the 
system** and evert himself, if need 
be, to get where he wants to go. 

Hfe rehabilitation Counselor; 
David G. Best'ot the York district 
office, calls him "supermotivatefl". 




John Foschia, right, ai work 

That he must be, for not only did 
he cap his ^rehabilitation program 
with a good job at the Compaction 
Division of Ingcrsoll -Rand, Shippens 
ourg, but he's had a couple of good 
raises in, the year he's been working 
t^iere. But John worked for that^ 
payoff. 

Mobility Problem 
When some yCars ago he entered 
Chamberstmrg Area Senior High 
School, he quickly found the cduca 
tional system of that time would 
majte no accommodation for people 
with mobility problems. If he were 
to get an education, he'd'hayje to 
travel the corridors, the sO^irs, and 
the campus like anyone else— but 
on crutches. AsMris course ncared 
completion.^John's guidance counse- 
lors referred him to the bureau's 
York office so that he might be 
better prepared to enter the world 



or; Employer 
Handicap- '• 

of work. With his rehabilitation ' 
counselor, John'soon ruadc plans- to 
attend the Pennsylvania "Rehabili- 
tation Center, how the Hiram G. 
Andrews Center in Johnstown, 

• Pennsylvania. There, vocational 
evaluation suggested he would do 
well in a business or clerical field, 
and he entered the training ^fimgg 
in accounting/bookkeeping, com- 
pleting inn May> 1978. 

Returning home to get on. wijh 
the next step — looking for "a job — 
he was dcla>ed 4 by trie need for 
further^surgery and recuperation in- 
^volvjptg a weight-lifting program at 
thefY.M.CA , to improve motor 
skills^ Meanwhile, his counselor was 
talking to employers wfio might be 
interested in someone with John's 
skills when he was ready. 

Employer's Role 

One of the 'employers contacted 
was Skip Remson , Personnel 
Manager at Ingersoll-Rand. ->$kip 
had been interested enough in hiring 
the handicapped to attend the 
bureau's 3-day seminar for employ- 
ers. "Understanding and Interviewing 
the Handica'pped Job Applicant" 
at the Hiram G. Andrews Center.^ 

Impressed fey what he'd learned, 
he'd returned home to consider how 
tngersoll Rai\J might productively 
employ sever* y handicapped people. 
Ingcrsoll Rai i. Occupational Ser* 
vices. Inc., md BVR cooperated 
m doing a >b analysis ol several 
worlr*arcas ir the plant. From this it 
was determme'd that , the parts 
.control area might be a promising 
area for developing a job where' 
mobility was iot a big* requirement. 
When a job opening did material 
ize there, O S.I. and the bureau 
wcjre able, to "retfer several prospec- 
tive* handicapped persons. Dave 
Magce, Personnel Adrriinisirator at 
Ingcrsoll Rand, who interviewed each 
person, says he could have Hired 

* any of thcrr because he felt they 
could all do the job. 

When Joh.\ Foschia ivas the one, 
hired, it caine as no surprise to 
Dave Best. 



Bureau Develops 
"Similar Benefits" System 

BVcR's .Evaluation Section is 
developing a formal system to 
collect, record, analyze, cvaFuate and 
"dissemirfatc information on "similar 
benefits". The term refers to ser* 
vices available to clients through , 
collateral agencies serving the handi- 
capped and which are use4 before 
bureau resources are tapped. 

Altoona Stfcjdy 

A pilot stuciy in ' trie* Altoona 
district office vvill collect informa- 
tion on all open cases, whether in 
diagnostic or accejpted-for-services 
status It will catalog all similar 
bepefits available to those clients 

0 for the diagnostic, physical restora* 
tion, . maintenance and training 
phases of their rehabilitation. The 
study will identify the sources of 
such, benefits and estimate the net 
savings that will accrue to the bureau 
through their systematic use in its 
15 district ^offices and the Tiiram 

, G. Andrews Cefiter. 

Policy ' . v " 

The i bureau has been insisting 
that clients avail themselves of simi- 
lar benefits since shortly a(ter the 
procedure was mandated by the' 
Rehabilitation Act of 1973. It re- 
.quires, for example, that clients in 
training ^apply for federal Basic 
Educationai^Opportunities Grants 
(iBEOG), "^Pennsylvania Higher 
Education Assistance funds 
(PHEAA). and otTier available 
sources before buieau funding is. 
considered. Similar benefits^ are 
available for other aspects of clients* 
rehabilitation, and£by standardizing 
and modifying benefits and their ap- 
plication, the bureau expects to 
maximize their use. 

Heading the program is Program 
Analyst Andrew Chopak. under the 
direction of Harry Guise, adminis- 
trator of tbe evaluation section. The 
a project is headed by Wash- 
^district administrator Frank 




BflJHos. with Region III Program 
Evaluator Paul Saupp as resource 
person. 0 
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• - * ' ; ** * ' APPENDIX 23 

PROFILE ANALYSIS. TECHNIQUE • * 

The statistical *&nd psycho logipal methods of expressing normal dis- 
♦tribution are jointly chaired by*both psychology and statistical text- 

7 * 

books. The normal, curve distribution' is a method of Comparing^data * 
through the. use of standard scores So that valid comparisons can be made 
of data, that follow* the ^normal curve distribution parameters. 

Historically, the normal curve distribution and'standard c # urves have 
been used by educators to grac^e-and make comparisons of , student achieve- 
merits. "The military haf also uaed these methods through the use of sta- 
nines iti,the Second WQrld War to test^and peasure military personnel - - 

achievement. . r * „, 

' >m \ ' " ' - 
The Pennsylvania General Agency pf the Bureau of Vocational Rehabili- 



tation, has chosen this method of making comparisons to meet- several needs 

, * \ - « ) 

* of the Bureau's program of serving handicapped clients. 1 

^ • . ■ 

First, we were searching for amfethod to equate the rehabilitation 

performance of our da^trict offices. Second, we were looking-for a mea- 

* / * * *« • 

sure to evaluate our performance with' other states and tfce nation (as re- 

quired by the nine Federal Evaluation Standards)* Thirds we were looking 

for a method to £how the strengths and weaknesses of our program. Fourth, 

we were searching for a^method and procedure that would increase our eval- 

uatidn* capactiy and effectiveness. Fifth, we would then utilize -their 

' increased knowledge to make recommendations for program improvements and * 

* ***** ' ^ ' 

accomplishments. . * * 

Profile analysis technique shpws on a visual profile where a state,* 

-region, or district ranks in program accomplishments. 

-+*- t r a m * 

Profile analysis technique, in the opinion of the author, has the same 

' ' • . ' . : l 

potential , for program evaluation as the original census data tabulating 

machine had for today's computers. 'It is a technique that is in ^ts 

"t- • " . V ' 

. ' ; ' " " i ' 298 
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1 t * 

infancy bdt has fc the potential to be used in any effective research f or ^ ^ * 
any* type of program. If the terms are defined and, the proper research^ . f 
procedures are followed, the profile analysis technique carl he utilized 
productively by any corporation, government, or person^ pi . * * 
Past, Present, dnd Future -z Program uses and accomplishments of rf the profile 
analysis, technique, since its origin in' 1976 in the Pennsylvania Geperal, 
can be'vieVed in r the ■ attached >Appendices. % * 



\ 



'151 



291} 



V ■ Profile analysis appendix ~ $[ \ \ 

SUMMARY OF PAT TECHNIQUES JJSED IN THE PA GENERAL AGENCY^ * 
• • • 

Past Uses \ * 

„,.l. Comparison of "data on the vocational rehabilitation follow-up surveys for 

Federal Program Evaluation Standard. 6 and 9. , ■ * \ 

' ■ ' ' ' \ 

2. Comparison of key performance factors among other vocational rehabilitation 
st^ate agencies . ; > 

3. ComparislSn of major disabilities at the district/regional levels. ' 
4* Comparison , of average cost for cases closed Status 08, 26, 28, 30.. 

- 5. See attached write up of priginal, ten factor outpat profile. 

■ • * 

Present Us€ » ' . t 

1. 'Eleven factor output evaluation and comparison of jregion and district perform-* 

ance. (see attached copies) ' 

* • . * • * • 

Future Uses ■ >. if" 
. — ^. 

'1. To utilize PAT in the formula distribution of the field offices case service ,; * . 
budgets . , ' t ' 

2. To implement a "Process Profile" to measure prqcessing variables such as " * 
-number of severely disabled accepted for services, and the average processing 

time in a particular status -or statuses. ' • ■. *\-*r?ffif: 

\* ~ . . • . 

3\ £o develop a "Referral Profile! 1 to measure input factors, such as- types of 

referral source and other demographic' information. , * 

• •., . / : , '. . ; * v- 

4. See attached copies of proposed new profile sheets and. latest write up of 

* " * ' i • 

pSrofil&s* rV- • ' " 

. * t The Evaluation Section wirl.be glad to furnish a completed copy of any profile 
uses for the present or 3 past uses, of PAT. - , v ■ <• ■. - < 
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TEST SERVICE BULLETIN 



V s 



T*t ceni of <out 
uneV po/iioni of 
iht no'rmol curvt 



Sjapdard 



Delations -4a 



Cvmvfotiv* Perc«niag«j 
Jtovnctd 



Percentile 
Equivalents 



Typical Standard Scores 
Mcwn L 



7»Kor«t • 



Cllt tcor«t L 



AGCT. tcor ts L 




Stonints 


. 1 


1 • 1 3 | 4 | , |. . | , | 


8 






9 


f*tr ctnt iryrtgnino * 

WecMer Scale! 

Subttitt L ► 


4% 


I -7% 12% 17% 20%. 17% . 12% 

1 • 1 " 1 I 

1— 1 i f 


7% 




4% 


I t 



Deviation |QjL 



1 

- I 

55 



10 



13 
I 



16 



70 



^ 85 



too 



115 



130 



19 
I 

145. 



\ve have included the lowest 2,3 per cent of cases. 
These percentages have been rounded in the next row. 

Note some otruir relationships: the area between ^fee 
drier points 'facilities, the scores which lie above the 
16th percentile' (-Icr) andlrtlow the 84th percentile 
(+la) - — two major reference points all test users 
should know. When we find that an individual has a 
$ ^8S£ ltf al)0v e4he mean, we conclude that his score 
ranlu at the 84th percentile in the group of persons 
on whom thi> tost was normcd. (This conclusion is 
gbod provided we also add th<$ clause, at least sub- 



vocally? if this particular group reasonably approxi* 
mates the ideal normal model. ) 

^the simplest facts fo« memorize about the normal 
distribution and the relation of the percentile system 
to deviations from the average in sigma,units are seen 
in the chart. They are " # 



Deviation from 
the mean 

percentile 
"^equivalent 



-2* 



-la 
10 



0 
50 



+ l<r 
84 



+ 2<T 

98 



The information reported for New. York, South Carolina, Texas and Pennsyl-* 
vania was extr<acted # .from Fiscal Year 1976. data for all BVR General 
Agencies iQ the nation* 

. • ' ■ y * • 

We have compared South Carolina to_New York, Texas and Pennsylvania be- 
cause of the significant increase in rehabilitations by. South, Carolina 
in Fiscal Year 1977. • ? • • ' . " ^ 



>* For Fiscal -Year 1976 in Seuth Carolina: 

"1. it was nqted that. 5^3 percent of, the total money expended was for 

r . .... * 

' - counseling and placement; while only 36.3 percent was for services , 

for individuals and only* 5 .0 .percent for administration. 

2-f, % That of t the tbtal' .expenditures feft: services for individuals, 50,9 

- percent was 'expended /for physical. and mental restoration and 2f£.4 

* Rercept for diagnostic and Valuation, and only 13.7 percent for training 

. 3. Thai- they ra'nke'd higher than New York,. Texas" and Pennsylvania -in: 

1 rehabs per Counselor; man-year (73) ^ ? • - 

w b. rehab Vuccfess rate (71) 

fc. severely disabled rehab success rate ,(63) 

d. non-^sevetely disabled rehab .success^ rate (75) ^ 

/■ e. /lowest average case service cost per rehab ($1,792)°' 

f. rfehabs per 100,000 population (427) 

* • *«, — 

With an increasing number of cases being rehabilitated by South Carolina, 

' • ' ' A > — *• . 

we, are*, interested in finding out how- they achieved their success. - Are the 

• factors leading to^the increasing number of rehabs* in South Carolina of a 

financial nature, is it the disability mix, is it the type of professionals 

employed*, or, does the^agency use other Existing facilities? ^ff 



Table 1. 



General Agencies 
Fiscal Year 1976 



Percent of Expenditures for Case Services 



diagnostic 
and' 

State .Evaluation 


Physical and 

Mental 
Restoration 


Maintenance 


^r^inirtg 


r ■ — 

Other 
Services . 


Post . 
Employment 


Nfcw Yoftc \ - 53798 17.7 
Pa. * ' n 30.15 13.0 
S % Car. ^ ' 86.86 25.4 
Texas . * 39.36 14.9 


9.6Q 1.8 
69.15 v 30.2 
96.49 50.9 
47.21 21.2 


56;75 14. 4* 
72.24 17-2, 
13^35 5.9 
•46.02 12. S 


93.94 62.3 
5}. 99 33.0 
5.82, 13.7 
61.41* 42.7 


^ core 
34783"* 3.7 " 

26*11 . 0.6 
35^94 4.1 
48.40 * 8.2 


~37*4V 0.1> 
33.*72 0.0 
33.72 6.0 
49.60 0.4 
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fable 2. 



General Agencies^* 
Fiscal Year i976 



percent of Total Expenditures for 



Administration 


* Counseling and Placement 


Individuals 


New' York 


25.46 | 


HI " 


12.92 


26.0.' • 


94.9T" 


67.9 


Pa. * 


31.56 " 




48.40 


35.6 


73.57 


57'. 5 


S." Car. 


17.89 % 


' 5.0 


( ' 98.98 


^ 56.3 ' 


7.93 


36.3 / 
44. ij-i 


Texas * 


44..43 


8.4 


89.25 


46.8 ■ 


27.42 . 
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Background Data for Paper 

^ * * * ** • j 

The, purpose of planifihg is for the improvement -of the performance, and 
* * . * - 

way of life in society, business, government \ and the individual's family 

> * - 

and dally* life. * ' • 

* v . ' — 

" J ' h M^ rpPSe ' ° f th . ia pap " on Planning and evaluation in the BureaU^of 1 
' '»4 ' ' .... 

Vocational 'Rehabilitation is to identify potential problem areas and deter- 

mine ways to improve them. The PAT mechanism utilized is 'flexible' and 

adaptable to changing priorities and client needs. ' 



The PAT (Profile Analysis Technique) is designed ,io_ work at the 
national, state, region, district and individual -counselor levels. It is." 
tied, in with -the implementation of the '.Rehabilitation >Act of 19737-Public' 
Lav 9.3-112, and the priorities of the .State. and Program and Financial (Plans 
mandated for each state by the Executive" Branch^ f the~Department of Health 

Education and Welfare. * . * 

t ^ — 

This,. Profile Analysis Technique, points out program deficiencies and 
.areas in .neetf of improvement. This paper, utilizes both the .PAT and SOAP. ' 
(Subjective, Objective,. Assessment Procedure) and is the First step in a 
three-pronged assault for -planning and program, efficiency! 
• The second step will be an objeetive profile of what variables 'effect ■ 
the end result < rehabilitation closures from-the time when a person becomes 
a Bureau client until they are successfully closed from the program). ' 

. The third step will be an examination .of the areas from which we secure 
our clients. This will enable the Bureau- of/ Vocational Rehabilitation to 
examine, our most efficient and productive areas andmethods and put them to 
use in other areas 'of the" state where we- show a need for program"' improvement 
The end re«ult^'of this three.-pronged planned attack for program improve 
ment will be the best attempt that the Bureau of Vocational Rehabilitation 



. - can. make to insure the survival of, our agency and make the future/less . 
uncertain. v * ,-•>•" i / 

<. . . -INTRODUCTION 

The^earch for .a Method to Evaluate Objectively ^e Subjective 'Factors of 
■ a People-Oriented ^ocatiWal RehabilitaVion Program - X - 

^ With, the advent .of 'the electronic computer," the mentation of large' . 
groups of data have Se come 'an objective reality:' 

Tl^ Tur^of Vocational Rehabilitation hWbeen searching for 
methodology to bring V 'the esu^ess factors 4n. a people-^rienteV rehabili- 
tation program together in. a data-oriented- V^.tiVe , stlnda.rd . # '. 5 

The presence .of data/ even L abundant quantities Joes not grantee ' 
a program, of objective reality. The ' peS ? t^o arV^so^ia^with the- * 

rehabilitation program generally hav*> a- „J$L'k -i> % '~ " . - 

F * generally have ^ psychopathic^ reaction to data man ipu 

lation. These people we. shall oatl-^^^/^. *\ • * ' " ^ 
^ The fac\ is that ^^io^^^^^ people are' sThoolsd in 
massaging and manipulating da'ta, 'These people we can call Researchers . 

Training, and Selling . V* * 

. The passage of Public Law 93-112, ^Rehabilitation Act of 1973, man^ ■ 
dated the setting up of Wuatiof Standards' of aJLl the Renabllitation / 
Agencies in the United States ., .What is netded to move' rehabilitation for- \ 
ward in the nation is ^ standard objective^.set of standards! 'These stan- 
dards must be so objective that* the reluctants can be trained and sold'on . 
their objective value. Without this training and selling job any standards * 
will be worthless., ' . , 

The utili ? a\ion of .people and succe.ss-priented< program data, with which 
th« field people 'are familiar, is essential -in getting up norm-referenced • 
standards. . 1 - 



- * In^.the opinion. of the Pennsylvania General Agency Program Evaluation 

' # ' 4 Staff, these standards should have the following "characteristics : 

> i , 1. Rehabilitation oriented. 

2. Simplicity in -computation. y 

4 ^ % 3.. Ease of interpretation. 

" • 4: % Flexibility. 

5. Multiple variable and analysis. 

.. * 
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SECTION I 
Profile Analysis Technique 
, The , essential selling point of this profile analysis technique is 
its ability to meet the five crS-teria listed. The information available 
in the multiple form used to collect data by State 'VR agencies, as for 
^ ^ example the BVR-300, can be used to set up standardized national or state 
"standards. The information collected "nationally can be compared and 
standardised with other spates. The information collected statewide , _but " 
not nationally, can be used to establish Bureau stanSards. In each case 
y^thxp information- can be standardized^so th^ the information can be analyzed 
% under a standard ^format at th£ individual counselor level. 

Examples: s * . *, 

Tcf get familiar with the mechanism of how PAT operates, turn to 
Example No, 1 on our example pages. It details a profile of- State Agency 



%k» 0 (simulated data) on several significant factors and compared^by five of 
<these#° As you examine this profile you wili see that the rows have num- 
bers ranging from 1 to 9 and- the columns, are alphabefcic'*showihg the .program 
variables being evaluated, e.g., (1) number of -rehabilitants ; (2) rehabili- 
tations per J.0p, 000 population;, (3) averagg rehabiiitatipns^j^ef counselor; • 
(4V percent of total cases closed rehabilitated that are in/ competitive 
employment; .'and (5) percent of total rehabilitations that are public welfare 
rehabilitations. - * 

As you* loojc at the left-hand side of the profile, note the shaded area 
that starts at number 5 ^nd continues Across the center of the chart. In 
PAT 1 prof iies thi y s represent^, the mean or simple average* As you move up the 
chart ye^T move above the average. As you move down the ch*art, you of course 

* move below the 'average. The range distribution of the chart is as follows,: 
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. Profile scores of 9, exceptionally high 

Profile scores 6f 7 and 8 are. high _ 

Profile scores from 6 to 4 are in average range 

Profile scores' of 3 and 2 are low 
. A profile score of 1 is exceptionally low 
p With this background^ we would interpret the profiles presented on 
our example pages, as referred to earlier, indicating an agency^ who^e , 
(1) total rehabilitation production was high based on national standards • 
However, their production of (2) rehabilitations based on a per 100,000 
population basis was average while the (-3) rehabilitations per counselor 
were high, (4) the number of cases closed rehabilitated in competitive 
employment was high compared to national standards and (5) percent of pub- 
lic assistance rehabilitations was high. 

As' the State Director evaluating this profile, *you feel a sense of 
pride at your agency's accomplishments . One of the reasons why your agency 
has achieved this standing is your constant evaluation of pro£*£tn progress 
and consistent search "fop: improvements. » 
p -A second example of PAT applied: • 
• If you look at Example No. 2 you will see another example of a profile 
of agency performance.- Here you see (1) high number of rehabilitations; (2) 
h£gh rehabilitation perN.00,000 population; (3) high rehabilitation pro- 
duction per counselor; (4) percent of cases closed rehabilitated in competi- 
tive employment was low; and (5) low percent of public welfare clients 
rehabilitated. The contrast between the two agencies is apparent by the 
production thrust of the second agency compared to the overall combined'f ive 

factor teamwork thrust of the first example. 

• „ Ease of 'Interpretation » * % 

1 The key, Selling point of the PAT Analysis Technique is one of simplicity 

**** . » *■ * 

and standardization of interpretation and communication. Take for example, 
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the two contrasting State Agency profiles we just looked at. The 

differences wei^e able Xo be evaluated because the performance o( the two 

* 

agencies was referred back to a common standard stahine scores ranging 
from 1 to 9. With the adopting of PAT 'five is always the average, qine 
is the highest score and 1 is the lowest score possible. As you see, 
.ease of interpretation and communication are achieved' with PAT. t The inter- 
pretation of these pro-files is also an exercise of ease and simplicity. 
Contrasts are made possible by transferring the information from paper to 
transparencies. Office machines can be utilized which do most of this work 
in a matter of seconds. The superimposing of one transparency on another 
to show contrast in tire profiles is quite convenient. 

RehabilitatioiKOr iented 

The Pennsylvania General Agency feels that these standards, to be 
meaningful as they are developed, must be rehabilitation oriented. This . 
metrfis that the models built "for program evaluation of the rehabilitation 
structure must include the familiar, structure functions, processes, and 
jargon of the field service ' staff and case service management. Everything ^ 
else being equal, standard^ should be related bacfk to these structures 
and processes . « 

At the counselor , ..client level many advantages can be obtained by 

demonstrating a PERT view of rehabilitation. PERT can be defined as a 

A 5 

manager's tool fpr spelling out and coordinating what must be done to 
successfully accomplish the objective of a project on time. Another view 
is-one of the computer technology or ,input, output system. Rehabilitation 
is not, however, a computer. It is dealing wittf people and, as such, is** 
not subject to a profit-making standard". 

The five standards, we iiavt identified for comparison on the two sample 
state profiles make it clan that tho structure of PAT is a rehabilitation 

4 
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oriented approach . What we are trying to communicate on this profile 
(PAT) by leaving 5 spaces blank is that other factors needing standard i- 
zation are easily added to the profile. In ad/dition, there is nothing 
•sacred about the 5 factors listed. They can be amended, changed or 
altered in ar\y way deemed desirable. 

» 

Flexibility 

The .third characteristic of the PAT system is its flexibility. Not 
only can we compare ourselves with' national and regional norms, but we 
can compare ^selves to other states, or comparable areas within our 
state with the state norm. 

The norms for a region within a state could be superimposed on a PAT 
Profile Chart as in Examples 1 and 2. • 

In addition, a counselor's performance could fce presented, showing 
his performance against another~counsfelor in his office, region, state, 
another state "or the federal region or.the nation. 

. With a data base established for production, a counselor might score 
a 6 in his performance in his office' compared to similar carrying caseload 
c^nselor's district norm of 5. Thev^ndividual district office couu*elor 
or district average can^be compared 'to the state agency horm. The same 
counselor productivity may be > the equivalent to an 8 on a regional basis 
and a 9 on a national level. 

4 

Thus the emphasis on flexibility of this profile technique. 

Ease and Simplicity of Computation 
A must characteristic of any national standard is computational sim- 

Plicity. Without understanding,, no system can win individuals' acceptance 

- f ' a 

and backing. The PAT system by limiting standard integers from 1 to 9 

. goes a long way toward achieving this simplicity. ^ 

The PAT approach makes .use of the utilization of the numbers 1 through 
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thus'mak'es it easier to store and recorcf bits of information on the^com- 
puter file. In addition, the mathematical calculations required* fn &or- 
relation studies of squaring a number is made easier by single dumber factor 

r • 

Multipl'e Criteria Analysis % 
T^iis statistical evaluation »tool is one of the most needed aspects of 
PAT. Here we move v away from* the single variable factor thinking. We k 
approach the rehabilitation program as -a many sided entity. It is inappro- 
priate to ask: Is this program good? So many other issues regarding the 
values of the program need to be evaluated. The- questions to be asked are, 
Good according to what standards? ' Good according to what criteria? 

In PAT we bring together all these dependent and independent variables 
*>f the standards and examine their multiple factors as they interplay -among 

themselves to giv^us a multiple criteria view of rehabilitation at work. 

r 

Conclusion 

That PAT is not going to* turn rehabilitation into Jermy Benton's "Social 

Utopia", is apparent but what also is apparent is that we have added a new 

tool for rehabilitation analysis that is going to do much toward acceler- 

' i -j 

ating the progress of a good program into an excellent one. 
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SECTIQN II 

This section defcls with 1 adoption of the PAT Profile Analysis with 
the needs -and evaluation critexiJ familiar to the Pennsylvania General 
Agency. ^ , % ,/ ^ 

Background Vdta 
Bureau of , Vocational Rehabilitation for the Pennsyl- 
vania General Agency'is set up on a^State and District basis for evalua- 

tion. purposes, the profiles/ shown will be in State and District Office 

* ** 

basis listing a 10-point evaluation criteria. I 

' \, 

Second, since the administration of the District Offices and the 
State is. divided into^ regions , each Regional Administrator supervises a 
number of District Offices. The Rehabilitation Administrators are 
res pons 1 ble to the Director of Field Operations » 

Third, the godls of the Bureau are set by the Bureau Director in 
coordination with federal priorities and the State Boajrd and J spelled out- 
in, the State Plan ^and the k Program and Financial Plan each year. The 
rehabilitation goal£ set the'n become the responstbility of the Assistant" 
Director-of Field Services (as the Chief Salesman in the Bureau) to achieve, 

Fourth^ the terminology of the Bureau staff and technical expertise 
which they possess is geared toward a people-oriented rehabilitation pro- 
gram. The Field Operations .and Bureau staff are also most familiar with a 

i 

# "x 

tabular and percentage type" of data presentation. 

Fifth, the Evaluation Section ih cooperation with the Field Service 
st'aff therefore made the following adaptations to the PAT profile so that 
it .would be more acceptable to th^ Jfield and counselor staff. 

This resulted ir^/the stanine scores being converted to z-scores whfch 

in turn can be converted to percentiles, which the Bureau staff are familiar- 

witfi. The procedure for i conversion- of stanine to z-s.cores ^nd the 10-points 
» 

• 1 - 
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that make up -the Pennsylvania General profile w^ll be explained in this 
Section. 

A. . TEN ITEM EVALUATION PROFILE 
P 1/ l. Rehabilitations per Counselor Man-Years 
p 2. Success Rate for* Case's Closed from the Active Load 

26,28,30 

Q 2/ 3. The severely disabled success rate 



26 

r . 26,28,30 . 

p A. The non-severely disabled success rate 

26 

26,28,30 

Q. 5. The percent of rehabilitations closed in competitive employ- „ 
ment 

g ,6* Average cost per cases rehabilitated ( inverted) . case service 
cost ' J 

T> 7. * The number of rehabilitations per 100,000 population 

Q 8. The percent of total district rehabilitations ea):h district * 
closed as fefeverely disabled rehabilitations . w- 7 

q 9. The success rate for all closures 

.-26,28,30,08 / 

Q 10., The percent of total district rehabilitations each district 
closed as / Public Assistance Rehabilitations 

1/ Production Factor 4 J 
2J Quality Factor 5 / 

Economic Factor^ 1 ( ^ * m . 

These" ten rehabilitation variables were used to measure and evaluate 
rehabilitation progress because they combined the federal emphasis on 
-quantity, quality and economic etist. In. addition they: 
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1. Emphasized federal and state pnorifies^ in State Plan and * 
Program and Financial Plan. ^ r 

2. Were familiar te*ms to uehabilitatlon-oriented Bureau straff. 

, * - . * * ' 

3* Seemed «to have been the best' indicators of success under limited 

** 

financial .conditions over the period of .the Bureau operations. 
Both from an objective data interpretation ar^ from a consensus 
s pf Bureau staff expertige^utilizing the Delphi approach. 
Item A. Counselor. Man-Years ^ ~ 

1. Definition 

Counselor m^an-years is defined as. all caseload carrying Qoujiselors 
or trainees having a caseload, less supervised cases. ' * 

2. Method used to calculate ' 

Example: District A . 

1. Counselor A 10 man-months 

2. Counselor B . m 6' man-months 

3. Counselor C 12 man-mbnths 
4 • Counselor D 8 man-months 

5. .Counselor E 12~mafl^months > 

,6. Counselor F 12 man-months • 

7. Counselor G ^12 man-m0nths ^ * 

♦ 8. Counselor H 11 m^n-months /* 



*SL Counselor I 9 mah-months 

10. Counselor J ^ ^ 4 8 manr-months 

11. Counselor K 10 man-months. 

Counselor man-rmonths =* - 140 ' 

, = 9.17 

Total counselor man-years district . v 

# . b Total number rehafrs v 600 m g- 

7 Number of ipan-yeafs 9*17 * 
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3. Method used to calculate z-scores ^ 
State Man = 50.00 * * 
Standard Deviation =14 « ( 

* mm 

z-score Formula = z = X-X , f X ~ your score 
/ * CT *X = mean • 



0 



- = sigma 



. - ; % \\ ^ z'- 6W0. 17 ^ ■. 
! z = 1.21 Go to Table 1 



and convert to percent down left hand column to 1.2 then over t 

T01 = .3869. Since your score 67 is above the mean $0 you add it 

to mean % 3 « 

* . .5000 
+ .3869 

,8869 f ^ 
Go to Table U and convert .8869 to 89- or stanine 8. 
Plot tfi^^stanine on your* profile analysis technique chart. 
This shows you how you compare with the other District (Tffices 

in the State fot rehabilitations per counselor man-years. 

< * 

If your score had been 47 you would have followed the following , 
steps. 



z = 



x-x" 47-50 _3 • -.21 

14 : 14 



Since your score is a minus. "you subtract from the mean 



i . .* * On Table I point -.21 = .5000 

I * -.0832 



0 



On Table U .41681= Stanine 5 which you would plot ^n your PAT" 
chart. To improve your counselors must 'rehabilitate more-cases 
per counselor man-ryears. This factor is a production-oriented 
factor. ^ 4? 
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TABLE II 

CONVERSION CHART FOR CONVERTING PERCENTS TO STANINE 











Stanine I 


= 


up to 4% 




Stanine II 


= 


>from 4% to 11% 


- 


Stanine III 


= 


11% to 23% 




Stanine IV 


= 


23% to 40% 


-* 


Stanine V 


= 


40% to 60% 




Stanine VI . 




60% to 77% 




Stanine VI I* 




77% to 89% * 


» 


Stanine VIII 




89% to 961 




Stanine IX 




96% to 100% 





Stanine = 5+2* j-^y V = 



x scores 



x-x 



' 0' 
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SECTION ' III 
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\.- • SOAP: Subjective, Objective, Assessment, Procedure 

The Pennsylvania Bureau of Vocational Rehabilitation now has a^pb- 
jective assessment procedure for use in* maintaining its national leader- 
ship in the serving of handicapped clients as mandated by Public Law 93-112, 
the Rehabilitation'Act of 1973. 

The Bureau of Vocational Rehabilitation, it! its efforts to' improve 
program operation in serving the handicapped, has developed a methodology 
where all data variables used to evaluate rehabilitation performance can be 
combined in ah objective performance profile. These performance factors 
used standard scored so that one can objectively assess and evaluate th« 
strengths -and weaknesses of Bureau operation'. 

The advantages of this particular approach is that it gives the Bureau 
the ability to evaluate, in a scientific fashion, performance factors not 
only within - the state but between states and on a 'national level. In this 
respect, th^n, administrative' and procedural decisions are based upon 
scientific principals and 'f^ts flot upoh subjective data and interpretations 
* ' Base Year 4976. to Pre-sent 

1. Raw Data « . ^ 

2. Profiles 

a. By district 

^ b.\By Data Variable ^ 

- \ J 

c. State Composite Score 
c# * Summary of .Prof ile Progress (Impact) 

The principal accomplishment of this profile w4s a composite state im- 
provement for eightnprofile items over base year 1976. The percent of 
public 'assistance cases rehabilitated to total rehabilitations remained un- 
changed at- 25 percent of total rehabilitations and the average cost per 
rehabilitation vent from $870 per case to $926 'per case service cost. 
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19-76 


,■ '1977 


49 


53 


65 


69 


59 


65 


70 


• ' 12 


69 


71 


179 


189 


27 


31 



^The main factor behind this increase in average cost per reha}>ilita- 
tion wa-s inflation and the rise in the number of ^evef^ly disabled clients 
rehabilitated, which increased from 37.9 percent in 1976 tfr A3 .2* percent 
in Fiscal 1977. The cost of serving. the severely disabled clients has * 
always been higher fehan'for serving the non^-severely disabled. 

Impact inflicators of the' other areas of the program improvement were 

follows : \ 
Pennsylvania General Agency 
1. Rehabilitations per Counselor Man-Years 
2.., .Total Rehabilitation Success Rate 

3. -Severely Disabled Success Rate 

4. Non-Severely Disabled Success Rate m * 

5. Percent of Rehabs in Competitive Employment 

6. Rehabilitations per 100,000 Population 

7. , Total Program Success Rate 
The composite scores for the district offices showed that &11 districts 

fell within the average wft^fcall ten factQrs of program success were 
weighted. See attached copy composite district scores Which showed that 
Altoona had the^highest composite scpre* 

In Rehabilitations per Counselor Man-Year three district offices in the 
low or below average range indicated a need for program impxovsment or evalu- 

9 

ation to indicate .atypical reasons behind this. In the Phi-ladelphia District 

40* • . 

Office the high number of public assistance cases is one of the reasons why 

t^ie production per counselor man-years is low and the large number^ of 

severely disabled cases rehabilitated is another. The important point to 

note is that the composite score of all ten factors listed shows that the 

Philadelphia District Office fell well within' the low average s&Rge when 

compared with the composite scores of all the district offices. In the sue- 

cess rate for cases defied -from the active load, two district offices moved 
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from the below average range .into the -average. One other, district 

4 - 

'dropped .from the average range to the below average range. The point to 
remember here is that district office progress or lack of progress is 
shown in a comparison scale. A district may improve its raw score as 
shownvon^the percentage t scale a^id f sti41 drop in its performance scale be- 
cause it did not improve as much as the other district offices.*. The Allen- 

town District Office had a large number of severely disabled^clients rehab- 
i- 
litated and this was one of the prime reasons why its success rate was TLow 

both from the active load and £or the total success rate. The performance 

of the non-severely disabled cases as to Votal success rate shows a need 

for program improvement in this area for the Allentown District Office. In 

the evaluation of. the number of rehabilitations placed in competitive employ- 

^ ment, the Philadelphia District Qffice was low and shows a need for program 

improvemervt; and also the DuBois District Office. 

One significant factor in average cost per case was "the fact, that the 

*Jdhns*town* District Office had much higher cost per case because of the lar^e 

number of cases in the Pennsylvania Rehabilitation Center. The Washington 

District Office case qosts were also higher because of the large number of a 

college v cases . The Pennsylvania Rehabilitation Center handles fc a large, num- 
*\ 

ber of meotal disorders and other high medical and physical restoration 

-\ - * 
cases. * The cost* of housing the BVR clients in t{ie Rehabilitation Center are 

also included while most of the ot'ffer rehabilitation centers in the .state 

have a majority of clients who reside at home or elsewhere an4 therefore 

^housing costs are not included. ' 

* % 

* , 6 < . 

In rehabilitations per 100,000 population the profile showed the need 

* * r ■ * *" . 

- ^ *•* - t, * 

n sfor additional counselors j.n the Roseroont and Philadelphia District Offices . 
, * to s^rve all the handicapped clients in those ateas*. 

In the number of severely disabled clients as a percentage of total 

' * > * • 

ceases rehabilitated, the Wilkes-Barre Office was low and the profile pointed 

: ^ " y 177 , 



'out a need for this, district' to serve more severely disabled clients. 
The success ra"r« of all cases closed from referral and- 'the active 
caseload shows that Philadelphia and Rosemont are .helow average in 
having success with their clients closed from referral and the- active 
caseload. The primary reason behind this appears to be a large number Of _ 
public assistance client? in Philadelphia and also the' number of severely 
disabled clients. The below average success rates -for Rosemont District 
Office appear to be caused by the large 'number of clients that are placed in 

competitive employment. », 

Synopsis 

-The Profile Analysis Technique points out areas of program .'strengths ^ 
and weaknesses to the program district administrators; it shows as an impact 
indicators as per rehabilitations per'100,000 population; and how well .the 
offices are serving the handicapped in specific areas of the state. It calls 
for action and recommendations for change when "areas of low program achieve- 
ment are identified. In essence, this profile has shotfn that on the PAT 
scale' all BVR district offices for the composite scale are doing about 
- equally-well. It does show, however, that- on rela^d items, each district. 
hajKareas where program improvement xan be generated in serving and ; employing, 
the handicapped population of the state.. ^ 
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Section' IV SUMMARY AND CONCLUSIONS 

The ten measures^used in the Pennsylvania General profiles are J>y no 

means all the measures available to objectively evaluate agency progress 

nor are they necessarily the best measures. Rather, they are the best 

measures which Pennsylvania has come up with as evaluation, criteria to 

initiate their SOAP Program, Will they stand the test of time or will 

they give way to other more valid and reliable measures of achieving a 

successful program. Only time, further study and evaluation will tell. 

They are however^ a first small step toward a better procedure of program 

evaluation aimed at concrete pro gram ob jectives and goals that improve 

, m ^ 

program operations and efficiency. - * 

The effects of this profile have already helped BVR.to improve its , 
efficiency from its- data base Sf operations in Fiscal 1976^ The results 1 
will show in the completed 1977 Fiscal \ear comparisons . The profiles 
are an evaluation -tool that require constant demonstrating of value. If a* 
district or a region finds any part of the ten-point profile to be in- 
equitable then they have the -option to. suggest- a more practical or reliable 
factor tq replace the one they object toi 

Finally the State Profile points out a need for a counselor profile and 

v 

a national profile both of which can easily utilize t#e stanine of z-scores' 
*but because of the .myriad of differences^ between the states .and the indi- 
vidual counselor's territory of clients dif f erent-prof iles^appear necessary 
to>measure the efficiency, of services' to clients at least at the initial 
stage' of the profiles' existence. ' 

* It appears evident that if this is a permanent arrangement spelled out • 

by the different needs of BVR clients in geographic, economic and social 

' ' '- - . * 

alreas. of .the county "then upon this base starting with a counselor prof ile -a 

method must be worked out for combining a counselor profile with a state . 



179 _ 

330* 



profile and a national profile.* All this requires a constant .evaluation \ 
and development of various variables that make up success in the field 

» 

of rehabilitation. To accomplish this the barriers of communication and 
coordination must be eliminated and the counselor, the supervisor, and 
the manager must work together*as akteam to develop a profile, in Vhich as 
a* working member of the team they each have input and faith" in its ability 
to be as objective a standard as possible in dealing with evaluating clients 
and vocational rehabilitation success* * , _^ 
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FIVE YEAR COMPARISON 
OF DISTRICT OFFICE PROFILE 
FISCAL YEARS 197 6-* SO 




Excluded from calculation. 
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COMMONWEALTH OF PCNNSVLVANIA 

May 5, 1981 

subject, Five Year Comparison of District Office Profile! FiscaTYears 1976-1980 * • 

Regional Administrators 
TOt district Administrators 

Counselors * ^ f 

from, Harry W. Guise, M Adminis-trator 

Per: G. Wil$fc&t sltffittat ) Statistician 

The' State afld District Office tables give you a brief synopsis of your five-year 
progression of profile progress. Each of the profile items, where applicable, can be 
compared from base year 1976. . 

rPhin> r i Ca11 ^ 1976 and 197r P rof iles Vasured* ten items/. The average cost per 
rehabilitation however, was excluded from the composite score. In 1978 (tone* items 
were added to Che original profile (excluding average cost pe^ehabiiltfSo ^ "mak ng an 
eleven item profile, these two items were Average Earnings per^erfabilitant and Ave?a2 
Increase in Earnings per Rehabilitant . These two new factors were a&ed tb refteet th! • 
financial improvement in the clients' earnings, or lack of it. Ta calculate the average ' 

by the total number of rehabilitations. To calculate the, average increase in earning 

rrom'tne co^r i0 H ZV?"-* 6 ^ rehabilitati °n* earnings af *eferfal and subtraft it 
from the -total rehabilitations earnings at closure. We then, divide this, sum by the 
total number of rehabilitations. For example", . • - ' . 

/ * t 

Client earnings at closure '.$2,500,000 * * 

•Client earnings^ at referral 700 'oQQ * . > ' 

Client earnings increase ^ $l,30o]oOO* ■ 

I 1 ' ' ' • 

\ Then we divide this earnings increase by the total number ol rehabilitations/^ . 



1,800,000 nn - . 

24,000 * m 7 *' 00 Average earnings increase * > 
between, referrkl and closure. 

PROGRAM PROGRESS 



theSfth'^raShnilr 6 ; 3 " ° leVen factors evaluated, increased for 

io«^ e P ercenta ^ ? f severely disabled* cases rehabilitated increased from 38 percent in 
1976 to 56 percent ln-1980. . T ~° P ercenc . ln , 

■ " * ' 

The rehabilitation success rate — r ~7^" * * „ 

in 1980# J 26, 28,30 cllmbed from 65 percteatr jn 197,6 to 74 percent 

- ^ 

fact^rs'increasin^f iS ° n ° f f3C? ° rS P resent V ^ed' in the profiles"" shows five ' 

pScal l^S! 8 * "fining constant and four factors' decrees ing slightiy from . . 

oa <, t R f?i 9nal ° istrict Administrators can Aview their districts' per formic*. Wr the 
Rast.-five years and compare it with the overall state performance 1 f> ' °.* 
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COMPOSITE SCORE 
Oct. 1, 1930 - Karch 31, 1981 
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AVQ. INCREASE IN EARNINGS PER REttAblLITAXT 
Oct. 1» 1980 - March 31, 1931 , 
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S. D. SUCCESS RATE (26+26+28+30) 
Oct. 1, 1980 - March 31, 1981 
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AVERAGE EARNINGS PER REHABILITANT 
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PERCENT <JF P. A'. REIIABS tO TOTAL REIIABS 
' Oct. 1, 1980 - March 31, 1981 ' 
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SUCCESS RATE (26+08+26+28+30) 
Oct.-l, 1980 - March 31, 1981 
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S. D. REHABS AS Z OF TOTAL REHABS 
Oct. 1, 1980 - March 31, 1981 
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PERCENT OF 8EIIABS IN COUP. EMP. 
Oct. 1, 1980 - March 3V, 1981 
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KON-S. D. SUCCESS RATE • (2^26+2a+30) 
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REHABS PER COUNSELOR KAN-YEAR 
Oct.- 1, 1980 - March 31, 1981 
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PROFILE ANALYSIS, TECHNIQUE IN 
THE PENNSYLVANIA'~GENERAL BUREAU QF 
VOCATIONAL REHABILITATION 



EVOLUtlON 



The Rehabilitation Act of 1973, Public Law 93-112, which was man- 
dated by the 93rd Congress, States that the Department of Health, Educa- 
tionand Welfare and the State Rehabilitation Agencies conduct studies 
to evaluate the Rettabiliation Program, These regulations were published 
in theFederal Register , '.Volume 40, Number 245 on Friday, December 19,^ 1975 
, and were submitted to the states in Program Instruction Memorandum RSA-76-39 
July 22,^1976 and IM 76-70, April ^27, 1976,., so that these regulations' be- 
came|jg|fective in the states in Fisfcal Year 1976, These instructions dealt 
specifically with* the nine Program Evaluation Standards, 

With the advent of these nine program evaluation "feandards came the 
need for a technique fcjiat could .be. used to make comparisons of Pennsyl- ^ — 
vania's data with those Df other states and the nation. This was respons- 
ible for the start of the present BVR profiles. 

Ten outcome variables* were utilized from 19^5 to 1977 and in 1978 Aver- 
age Cost per Case was* deleted from the composite factors and two new factors 
were added to make the present eleven outcome variables for the state and 
fifteen district offices. 

Ten Item Evaluation Profile (PA General State and f if teen^district offices) 

' ' + . ) 

P. 1. Average Rehabilitations per. Counselor Mari-Y&ar ' 

This factor was utilized to equate performance by counselor with 
time spent oh the job. In essence it was how many rehabilitations 
the district office counsfclors^ produced for the time workeck, 

P- 2, Total Rehabilitation Success Rate from the Active Load 26 



« " ■ " \, 26,28,30 

This measure looked at the number of cases accepted and successfully 
rehabilitated, compared td* the number of failures. * 



Q, 3, Severely Disabled Success Rat^ 



This measure was put into the profile to place added emphasis "on 
serving and rehabilitating the severely disabled' clients, and to 
measure their success rate compared to 'the non-severely disabled 
clients , ' ■ • v - ^ ' - * 

— . s \ 

Non-Severely Disabled Success* Rate 

* *■ ' ^ ' " ' ' • 

This meastire sJptfVs *the sugtess rate of the non-severely disabled, 

5 • Percent of Rehabili t ations in Compefi t ive Employment * 

This is a me.asure to plate emphasis on putting BVR clients back 
into thfc competitive work force* ( 
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: - - , - - . ■ -- " .- • fi *\ • 

B . 6. Average Cost per Cases Rehabilitateg (inverted) cases, service. 'A 

cost ■ . , * 

This item was to measure cbstrpf rehabilitation services by dis.- * 
trict office . " - ; . 

P. ?. Rehabilitations per tOO.OOQ population ' <* 

— This measure was utilized to show equity of services throughout > 
the counties in the .state and for". the- BVR profiles equity of 
services to the rehabilitated clients in the district offices. . 

q. 8. Severely Disabled Rehabilitations as a' Percent 'of Total ^ ^ 
Rehabilitations 

The percent of total district office' rehabilitations each district* 
office closed as severely disabled rehabilitations. This -measure 
was utilized, to emphasize the severely disabled clients b'elng rehab- . 
ilitated, because at the time that the profiles were originated W ' 
was mandated to expand services t.o the- severely disabled to 50% / ^ 
of its active and closed caseload by 1980. • - 

' " ■ ' 

0. 9. The Rehab Success Ra te for all ^Closures , . . ? , ,» , 

~* — 1 — ' ... ,/V 

This measure was placed in the BVR profile to emphasize the evatua 
tion process before accepting a client for BVR services. 26 ' 

26,28", 30,08 



q ft) fc e Percent of "Public 'Assistance Rettabs^'compared to' Total fefrabs. . 

This' measure was adde'd to emphasize the .priority the federal and. 
state governments put on -putting public assistance clients back to 
work, (not' homemakers) • • 
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Composite of Profile Factors __v. „L l .- «*- 



P = Production Factor 4 ; / ^ ^ 

Q = Quality' Factor . .' 5 . . . , 

E = Economic Factor j_l " , . « • - 

-.Total Factors 10 **•.'•«•''* V» 

These ten rehabilitation variables were Wd to measure and evaluate 
rehabilitation progress because the.y combined _ the. federal emphasis on 
quantity, quality, and economic cost. In addition they: • 1 

.1. emphasized federal and state priorities in State Plan and Program and 

Financial Plan . » . » a 

2. were familiar terms to the rehabilitation-oriented Bureau staff 
3 seemed to have been the^best, indicators of success under limited 
financial conditions over the lifespan of the Bureau operation 

Fistal Year 1978 marked a change in the BVR profiles. Two new items _ 
were added to the original profile (excluding Average Cost per Rehabilitation) 
. making an eleven item profile.-' Average Cost per Rehabilitation was retained 
as an indicator -of cost for the district office, but because of atypical sit- 
uations in. Bureau rehabilitation cost\ this factor was -not used in calculate . 
ing the district office composite score. 
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Two new measures jpdded were Average Earnings per Refiabilitant and 
Average Increase in Earnings per Rehab i lit ant « These two new factors were 
added to reflect the financial improvement in clients' Earnings, or lac^ 

* of it. The BVR profile now.had 2 economic, 4 production factors and" five 
quality factors for a total of 11 outcome variables* 

* SCOPE OF STUDY * ' 

The investigator in this study" takes into consideration the impact 
r that these eleven outcome rehabilitation variables have had' on thh program, 
first in each district office than the state and finally as a "single" com- 
posite score to measure their total impact on program efficiency. Each of 
the elevep program measures is also evaluated from its base line or origin 
'for its^ impact on program improvement^ efficiency and meeting program needs, 
goals and objective^ . * 

PROBLEM STATEMENT , * . 

An objective metliod^was needed to evaluate , all the subjective perform- 
ance factors that* are corrftined in making successful a people-oriented 
rehabilitation program, . ' 

This objective method was needed to evaluate and combine all the £ 
subjective component p^sts of each individual "objective performance success 
variable (four production' factors, five quality factors and two economic 
factors). Th^fe factors were then combined,; in an overall objective perform- 
ance standard*; 

PURPOSE ^- r , 

To express in one- score, a weighted index of program performance that '* 
measures the extent to which *the agency hai fulfilled the mission of the VR^* 
legislation — employment of the handicapped^ with primary 'emphasis on serving 
"and < rehabilitating the severely disabled client. 

The PAT (Profile Analy5i3~~Technique ) As designed to work at the national,' 
state, regional, district and individual counselor levels to show on a 
visual' profile, equal objective measures of program sucdess — rehabilitation 
,of the handicapped. - to * * m * * 

The PAT mechanism .was selected because of its simplicity in cdmputration,. 
e&se o£ interpretation, flexibility and it is mult'i-variable oriehted. 

* . .***.. 

v The Profile Analysis Technique is also adaptable to changing program * 

priorities and client*need& and identifies "potential problem areas and 

program needs. « . . 

PAT is a mechanism which enables the Pennsylvania General Bureau of Voca- 
tional Rehabilitation to push the priorities of the State Plan, Federal Prog- 
ram and Financial five-year plans. PAT can be ut^Lliz^ed in projecting program 
needs as mandated for each by the U.S. Department* of Education for Rehabili- 
tation Services "Administration ^nd for the State Budget Office. 

The Profile Analysis Technique is an ongoing program of evaluation re- 
search in measuring program progress and success. 

^ k ^ * 

0 * , 
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The analytical techniques measure,^ll impact indicators. Sociologi- 
cal:- *Tlow many clients can no* funS'tion independently in the home? 
Economic: "Hoi much of -an earnings increase did the client or clients have 
between earnings at referral and earnings at closure and statewide this 
had a minimum effect of adding $94 million to the state s economy .for 
Fiscal Year 1980. Cost Benefit: How many public assistance recipients 
were-rembved from the public welfare roles inFiscal Year 1980 and how • 
many dollars did "this save the taxpayers of the state? , The savings m 
public welfare cost was approximately $6 million n Fi-sca "80. Program 
Efficiency: How many disabled clients were rehabilitated? Rehabilitation 
per" 100,000 population shows how mar>y rehabilitations compared to the 
total population available and the 'success rate shows how many rehabilita- 
tions from the number of cases received. 

. The use of standard scores to show equality* of distribution under the, . 
normal curve (probability curve) was utilized through .the use of Standard 
z-scores to make- objective comparisons of program progress. 

RESEARCH METHODOLOGIES 

The research design employed in this analysis is essentially a com- 
parison of the eleven proven program measures of success measured from a 
base year 1976 (before) and program progress achieved at the end of the 
fiscal year (after). A weighted composite" of these eleven factors is then 
utilized to evaluate total district, region and State progress. Standard 
scores were "utilized to make valid individual parameter comppnsons and a 
composite eleven-measi&e score utilized to arrive at a wieghted comparison 
between State, Regional and District Of all peVformance measures. Data 
sources are the computer Clieaf Master File, which is a statistical file . 
of 128 items and 387 field positions artf a Case Cost History File of 30 
items and 120 field positions. 

METHODS " ' t 

Tho rp-h^in^inn n.it-p.rt- variables wereLselec-f£-d_£PX-the study by the 
•use of the Delphi Method. Delphi forecasting is a widely^used' technique , 
for the systematic development of expert_ opinion consensus concerning the 
future. This approach is in essence the utilization of -the expertise of 
the rehabilitation program administrative staff to come up with the most 
important factors in the 'success of the joint Jeder al-State rehabilitation 
pro grams . " > „ 



. The second technical approach- we utilized .was measures of central _ 
tendency, utilizing the normal .curve distribution. The eleven output vari- 
ables ,were then converted to standard, score's so_that the data variables 
could be compared for evaluation purposes-. 

The third technical approach we "utilized was a subjective-objective- • 
assessment procedure where we utilized standard scores *o combine the 
eleven outcome variables in a total "district and state profile. , _ 

" We analyzed the state's, statistical indicators of program achievement 
and from them selected the eleven proven indicators of program success and. 
"eas of service priority that were written into the .State Plan and mandated 
federal program priorities. The (fata wasTefived from the RSA-300, Case 
Service Report and the normal curve, distribution using z-s.cores was utilized 
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to" compare , and standardize "the data. The impact that we. were evaluating 
that data for, was the reason behind and the method that: caused success, 
for one measure in one district , + but relatively poor performance f<?r that 
same measure in ancrther dist*ict.vi The dat^a that was evaluated was for 
Fiscal 1976-1980, Computer programs were utilized to -secure the RSA-300 
data in the fabul'ar format necessity to work up the profiles. 

The PAT profile is distributed tq theSjtate, Region and District Off- 
ice Administrators on a quarterly visual profile (see attached example). 



Report Writing (Impact) ^ 

Program- progress over base year 1976 was achieved in the state composite 
raw percentage scares for each year from .1976 to 1980. 

4 The state composite, raw percentage' scores for eight of the original ten 
rehabilitation faci^rs evaluated increased over base year 1976. 



/rUe, percentage* .severely disabled cases rehabilitated increased from 
38 ^ercient in ■l r 976" > tp 56 percent in 1980- V 
— , , ' — . " . .■ ^6 . 

The rehabilitation success rate, of. 26,28,39 climbed from 65 -percent 
in 1976 £o 74 percent in -190., L The success rate for the severely disabled 
cases rehabilitated climbed to frfi percent . Impact indicators of 

other' areas of program improvement were^s follows: 

Pennsylvania General Agency *■ „ 
1. Refiabiiitatitfns. per counselor man-year 
2\ Non-Severely Disabled Success Rate 





1-976 


1980 




49 


53 




JO 


80 




179 " 


189 


total rehabs 


23 


26 




-27 


42 



3. Rehabilitations per 100,000 population' 

4. Percent o'f- Public Assistance rehabs to 

5. Total success rate 26 

26,28,30,08 

The average case service cost per casef rehabilitated went from $ 870 in 
1976 to ^$ 901 in 1980, an increase of four percent and considering the 
clouble digit" inflation of this, period and- ehe change in the caseload mix to 
severely disabled clients, this achievement was only possible through con- 
siderable team effor;t on the part of the BVR administrators^ and counseled 
'to control costs: For example, jconside/ the difference in case setvice cost 
-of the severely disabled rehabs vs. non-severely disabled rehabilitations \ * 
for Fiscal 1980* The severely disabled cdst per rehabilitation was $ 1,069 
compared to $ 690 for non-severely disabled cases rehabilit/ted\ The percent 
of cases in competitive* employment dropped s,ix percent f^rom 69 \o? 63 percent 
from £976 to 1980V primarily- because the Bureau was building up i^s ability 
to serve and rehabilitate clients whose prognosis to function in the compet- 
itive labor market , was not as high as the non-severely disabled clients. 
Please note that the. Bureau success rate, 26 , for cases for the non- 

,26,28,30 ^ 

severely 'disabled clients closed in competitive employment w^s 75 percent 
compared to 53.5 percent 'for seyerely disabled clients. The ^range between 
these two success rates was 21.5 percent indicating the increasing difficulty 
of placing the severely disabled in competitive, employment . 

In Fiscal 1978 two new items ^were added to, the original profile (excluding 
Average Cost per Rehabilitation) making an eleven item profile. These two 
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items were Average Earnings per , Rehabil itant and ^v^rfege 'Increase in Earn- 
ings per Rehabi li tant . These two new factors were added^to reflect the 
financial improvement in the clients 1 earnings or lack o? It. The Aver- 
age Earnings per Rehabilitant has increased from $97^ in 1978 to $ 103 in r 
1980, while the Average Increase in Earnings per Rehabilitant^ has risen 
from $77 in 1978 to* $75 in 1980. \ " 

The composite scores for the fifteen district of f ice s # , shotted' that all 
districts fell within the aver aget when ^1 eleven' factors of program suc- 
cess were equally combined.' See attached copy of composite district scores 
which showed that District H had the highest composite score. 

In rehabilitations per counselor man-year two district offices scored 
ia the low and one in r the*below average range. An evaluation of the primary 
reasons behind these low* scores is necessary to determine whether we have 
atypical circumstances in these districts or whether we just need mors 
emphasis "On productivity per counselor. In District Office A we find over 
40% public assistance cases and over fifty percent severely disabled cases 
as the primary reason behind the low production per counselor man-yea^. 
The important point to note is thaf the composite score for- al'l ten factors 
listed shows District Office A fell well within the low average range when 1 - 
compared with the composite selves of all district offices. In the success 
rate for cases closed from th$ active load, two district offices moved from 
th^ below average rang-e into the average, and one district from the average 
range to the below average range. The poi.nt to remember here is -that dis- 
trict office pro^pass or lack of progress is" shown on a comparison scale. A 
district may improve its raw score as shown on the percentage scale^ and, still- 
d roup- in its per formance ' scale because it did not improve as much ^s the other 
district offices. District Office C had a large number of severely disabled 
clients rehabilitated an'd this was one of the prime reasons why its success 
rate was low, both from the active.; load and for the total success rate. The 
performance^! the non-severely disabled cases as to total success rate shows 
a ne-ed for program improvement in this ,area 'for District Office C. 

In the evaluation of the number of rehabilitations placed in competitive 
employment; District 'Off ices A !and I were, low and show a need for program 
improvement. ' j 

One significant factor in average cost per case was the fact that District 
Office J haci a large number of cases in the BVR s tate-op^rated Hiram G. Andrews 
Rehabilitation Center located iri that district- This^rehabilitatlon center 
deals primarily 5 with /the severely disabled and most clients stay at the center 
when* receiving services,., ^District "Office Oca&e costs weVe also higher be- 
cause of the .large number* of c611ege cases. ' • « * 

In rehabilitations per 100,000 population, the profile showed the need for 
additional counselors in District! Offices *C and A to .serve „ all 'the ^handicapped 
clients in these -/areas. '* '* v* 

c In "the number of . -severely disabled clients' as a percentage of the total ^ 
cases rehabilitated, District Office F was low and the profile pointed Qut a 4 1 
need fo'r this district to "serve more severely disabled clients. 

: The success rate of all cases closed trom referral and the active case- ^ ^ 
load 'showed that District Office A and District Office B w£re below average in 
having* success with their cl ients 'clQsed from referral and the active caseload. 

2*02 
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The primary reason behind this' appears to be 'the large number of public 
assistance clients and severely disabled clients in Dis&r.ict Office A.' 
The J>e low ,ave rage success rates for District Office B appear to' be caused 
by the large number of clients that are jplaced^in competitive .employment. 

SUMMARY ' • > , , * • 

The Profile Analysis Technique pQint out areas of program "s trength 
and weaknesses to the program district administrators; it shows as an ' 
impact indicator the rehabilitations per lOO'/OOO population; and how 
well ""the offices are serving' the handicapped in -specif ic areas of the 
stated. It calls for action and- recommends change frhen areas of 4 low 4 
program achievement are identified. In essence, this profile has shown 
that, on the^PAT Scate all;.BVR* distVict%f fices for the .composite scale' 
are doing aboti,t /equal ;ly .we 11 . It does show,, however^ >£hat on related* 
items, each district* ha£ are 4s where ,prog"ram improvement ca.n be generated 
in serving and^employing the handicapped population of *£t\e *Stat/e . 

All this requires a constant: evaluation, and development s of- the- depend- 
ent and independent variables that makeup the s.ucc^ss in 'the f?.eld of/ , 
rehabilitation. To accomplish this' the, barriers of 'communication and 
coordination must be eliminate^' and' the counselor, the supervisor^ and the 
manager must work together as a team* toMevelop a .prof ilfe */in which as >a 
working member of the team each has input and faith i>n 'its Sbilj-ty to be 
as objective a standard as possible in dealing with' evaluating clients and 
.vocational' rehabilitation success. " 
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REFERRAL PROFILE OPEN CASES- 

Acceptance rate for public ref erral\sources , using refertal source codes 
14, 20, 32^ 34, 38, 40, 50, 51, 52, 53, 5>, 55, 56, 57, 58, 59, 80, 81. 
Acceptance rate for private referral sources using referral source codes 
10, 12*, 16, 19, 22, 24, 29, 30, 39, 44^-60, 62, 69-' 

Acceptance rate fpr individual referral source using referral source codes 
70, 72, 79. 

Acceptance rate for referral source code 50 and SSDI/SSI code 1. 
Acceptance rate for referral source code 50 and SSDI/SSI codes* 2, 3,* 4, 5. 
Average months in status 00 for all cases currently in status 00. (Inverted) 



ently ii 

V... 



Average months in status 02 for<all cases currently in status 02. (Inverted) 
Acceptance rate (excluding 08 from 00). 



10 or ^higher 

10. or higher plus 08 from 02 +.06 



The rate of cases closed status 0§ from 00. (Inverted) 

08 from t)0 



' ' (08 from 00)'+ applicants (status 02 

<* • 

The rate of cases closed status 08 frotflr 02. (Inserted) 

■ * „ * 08 from 02 

(08 from 02) + accepted 
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' PROCESS PROFILE MEASURES* 
Acceptance or eligibility ratg 

This is the number of individuals determined to be eligible for services. 

Cases Accepted (Status 10) 

Cases Accepted and Closed (status 08) 

Average processing time from 00-10, (Inverted) 

The average number of months from referral to cases accepted. 

Percent of plans written to number of cases accepted*. 

Average processing time from status 10 through closure. (Inverted) Average » * 
number of months f rom* acceptance to status 26 closure. 

The average number of months from plan development to plan completed, (Inverted) 
i.e., Status 10-14 or higher. 

Average number of months from referral to closure for.f6 f s. (Inverted) i.e., 
00-26. 9 

Average number of months from IWRP initiation to 26 closure. (Inverted) i.e.^ 
14 or higher to 26. , . 

Average current fiscal year cost per cases 'served 02-39 including closed cases. 

(Inverted) Status 02-39 total fiscal year expenditures 

^2-39 total cases seized with fiscal year dollars 

Rate of delinquent annua4 comprehensive reviews. (Inverted) 

District Office Delinquent Cases , 

Total to* Xev tewed each month + delinquent cases 

<r • • • ' ■ 

Average number of SD to total open, cases statuses 06-39. 

SI) Open Cases 06-39 

• , ' All Open Cases 06-39 

♦ 

Percent of accepted cases (status 10) working at referral. 

Cases accepted (status 10) with WIS, of 1 or 3 at referral 
All Accepted (status 10) cases 
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* FACTORS FOR SUGGESTED OUTCOME PROFILE 

Average rehabilitat jkms per counselor man-years. ' \ 
Total. rehabilitation Success Hate f rom j£he active load. 



26 



26, 28, 30 



J? 



\ 

Severely Disabled Success Rate. 

Percent of rehabilitations in competitive employment.* 
Rehabilitations per 100,000 population. 
The rehabilitation Success Rate for all closures. 
The percent of putfllc assistance rehabs to total rehabs. 

Average E arnings per Rehabilitant closed in competitive employment (WS 1 & 3) 
Average Increase in Earnings per Rehabilitant closed in competitive employ- 
ment (WS 1 & 3) . \ . 
Percent of Similar Benefit services use on cases closed. 

* SB- services 26 + 28 + 30 + 08 from 06 
AL\ services. 26 + 28 + 30 4-08 from 06 

•Success Rate for the competitively employed SSDI case$ with SGA of $70.00 or 

more per week. / J 

SSDI cases with SGA of $70.00 or more per week "_ % 

Total SSDI relTabilitations having a SSDI control coc^ of 1 of 9 
< - • 

Success Rate for the competitively employed $SI cases with SGA of $70,00 or 

more per week.^ , 

SSI cases with SGA of $70.00 or more per week ^ 

Total SSI rehabilitations having "a S31 control code of 1 or 9 

Percent of po"st-employment cases served in the district office. 

PE cases served 



Status 26 closures in previous fiscal year 
Percent of No Cost cases rehabilitated ($35.00 or less) 

Total No Cost cases Sfcatus 26 closures 
Total Status 26 closures 
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TEN PREDICTOR VARIABLES TO CONSIDER WHICOFFECT SUCCESS RATE 
AND 'ARE PRESENTLY ON THE COMPUTER FILE 



1. Sex. 

2. Education v 

\ 1/ • * 

3. Client -Counselor Relationsjiip^ 

4. Time from Referral to Service Initiation 

5. Total Number in Family: 0 to 

6. Age- - 

• v * 

7. (1) Urban, (2) Rural (Geographic area) 

8. Average cost per case 

' 9. Primary Source of Support at Referral 
10. Number of Services Provided , " ' 
1/ 1 ^Accepted for services 3 

» 

2 = Refused services Status 28 = -3; 30 = -2 

3. Failure to Cooperate 28 = -3; 30 = -2 

4. / Cannot Locate 28 » -3* 30 = -2 ' * . 
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To §efine program evaluation on a state ^evel it is very useful to identify 
-procedures for utilization of information to improve decision making in vocational 
rehabilitation program planning, monitoring and revision. (Handbook of Program 
Evaluation Studies, Michigan Rehabilitation Research, 1978) , This review of current 
state rehabilitation agency program evaluation studies further states that program 
evaluation studies can be divided into three broad ar$as: A) input studies which 
provide information for use in program planning, B) process studies which provide 
information for use in program monitoring, and C) outcome studies which provide 
^•information for use in program revision or change. In this article our attention 
will be devoted to ,the process studies in program evaluations studies. • - 

» A process study such as a case review system is concerned with t&e case service 
pattens within the rehabilitation process. Examples of the type of information 
9* " 



tbered from a case review system might be: A) delayed movement of clients' cases 



in the rehabilitation process, B) percentages of eligibility of served clients, 

and C) suitability of selection of clients 1 vocational objectives . This type of 

\ compiled data would provide the appropriate people in the agency's case service, . « 

policy anji planning sections a "nitty gritty" information base for change,' modi fi- 

cation, Uevelppment and implementation of^^licy and procedure. 

For example, we can determine through statistical data the existing time 

4 frame's in the movement of clients' cases&fzom prefer red status to eligibility 

statulf. If we develop a*$tandard of three months as an acceptable time frame for * 
W § * 

that movement we might fearn that in 20% of our cas&s there is^delayed movement. w 



With this statistical information cfs a base a case review could be conducted on 
the 20% delayed movement cases to determine xeasons? for the delay. An analysis 
^\ *of the case review ffndings^wpuld provide sufficient; case service information to 

■v 



" implement case service policy changes, iQpdifications k or , clarifications . 

Above all a cast review proce$s^should attempt to meep t£e specif ic -needs $ 
mf tm ^diMidual VR agency' using ^the proqpfs. In Pennsylvania we feel that we 



y* • % 

a-w ~ La. g ft *f i 



have a* structured case review process that does meet our particular needs and allows 
for input from many ievels . \ 

a 

# . The structured process begins with our field operations organization. The 

agency organization 'provides for four regional offices, each supervising the field 

- » ✓ ■ 

activities of several district offices. ' A case service ^valuator is assigned to 

v » - 

each of these regional offices. 'But though these regional case service evaluators 

» * 
are assigned to t&e regional administrators, the. central office Administrator of 

Evaluation maintains* continuous %&id direct contact with the four 'to coordinate and 

direct case service evaluation activities. This dual supervision and control 

. assures a thorough integration of staff and line activities in program evaluation: 

and provides for- the continuous flow of information to and from field activities?* 

- The attached diagram (Figure 1) describes the relationships between the 

Central Office, Regional Office and DJ^trict organization and the case review 

process. Utilizing this organizational structure, Pennsylvania's. Case Iteview* 

Process has the following objectives ; % — 

1. Evaluate documentation for State and Federal compliance. 

2. Identify the strong and weak ar&as in case service patterns. 

3. Evaluate the understanding of existing policy and procedures. 

4. Identify inadequate' policy >and procedure. ' , > 

5. Standardize, as much as possible, the t interpretation and implementation " 

4 of policy and procedure. / • 

6. 'Promote a positive concept of evaluation's role within the vocational 
m 9 

rehabilitation process qontext. * m 

• *■ • » 

With these objectives in mind the following system has been developed to A 

implement the Case Review Process in Program Evaluation in Pennsylvania. This * 

^system has been further enhanced by Pennsylvania being a model '-unit state. "Depending 

upon the Case Review and the circumstances the steps, outlined can be modified ~ 

(Figure 2 is a synthesis of- the' Case Review Process)' 4 ' • 
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Following the below outlined key steps, a reference to an attachment is noted. 
This attachment is an example of the described step. , 
J. Selection by Office of Administrator for Evaluation of Ca^e Service Area for 

Review* ' ■ 

t 

» * A . Obtain input from a representative sampling of District Office personnel 
concerning Case Service areas that might be reviewed and the objectives 
for such review. District Office personnel might be District Administrator , 
Assistant District Administrator and supervisors . C 

B. Obtain input from Central Office Management concerning the Case Service 
areas that might be reviewed and the objectives and purposes for suggested 
reviews. (See ^Attachment 1) 4 

C. A Through the above input crystallize and select- a Case Service area for * 

review and state the objectives of -the 'review. (See Attachment 2) 

D. Advise Case Service Evaluators of Case Service area to be reviewed. ; 
II. Development of Case Review Sheet and Standards . 

A. Each Regional Case Service Evaluator develops a list of appropriate questions 
to be used for the intended Case Review, and a copy of each list is sent to* 
each of the other Case Service ^valuators and to the Administrator for 
Evaluation. , 

The lists / of questions are reviewed at the regular monthly meeting of u the 
Evaluators with the Administrator for Evaluation - reviewed f of the purposes 
of deletion, addition, changes and the compilation of a single list. 
C. Following the above meeting and prior to the next monthly meeting each 
w Evaluator selects a few cases to review with the single- list for v. 
priateness of the questions , and again completes his or her' own 
which again is sent to each of the* other Evaluators and to the 
' Administrator of Evaluation for review. «, 
, D. At the second meeting the Evaluators' again review the, questions together 



9 

ERLC 



2l6 * 

36.6. 



/ 9 

and complete a final list of questions with standards, with categorization 
of areas of prime importance, using, if possible, an existing base of 
information such as R-300. • v 

E. Following the second ^meeting each Evalugtor .pretests' the final questions 
and standards by reviewing 25. cases of his or her own* choosing . 

^\ } 

F • At a third meeting^the list of questions jdsed in tfre 15 Case Reviews are 
> , ' p , \ * ^ / . • - " 

-distcussed. A question i€em^nalysis of the. Case Review Shebt is performed . 

for uniform interpretation and increased interrater reliability • . (See 

• " T ' 

attachment 3) . 
, G. The Case Review She^t^and Standards 4 are drawn up and completed by the .group 
as a whole at^this third tneeting. , 

The tinal draft of the Case Review Sheet and Standards is' presented txx 
the appropriate Central Office management level for its review and inpotv 
and to a representative sampling of Distrigt Administrators for their 
* review and input. * 5 , 

• X, Office of the Administrator of Evaluation finalizes the Case Review Sheet 

r • 

and Standards. (S&e Attachment 4) 
III. Cage Review x » ' 

• ... « • ■ . ■ 

A. Meeting of Regional Case Service Evaluators to Reinforce ' Interrater 
Reliability. * " t \ 

1. At a regular "monthly meeting bf the Evaluators, or a Special .meeting 

4 ■ called for that purpose there 'is discussion of interpretation of policg - 
- and procedure, * ' ' 

2. Standards >a$d Case Review Sheet to be used are thoroughly reviewed*. 

3. ^valuators rjsvi^ty a case with* the Ideyeloped Case Review Sheet and , * 
Standards/ • , ' *• 

' 4. Question interpretation /or Case Review Sheet is> developed. 

Selection Jby office of Administrator of Evaluation of*'Cases to be Reviewed-* 
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' f0 l 2. Computer is used for selection of r^nd^m and stratified Sampling' of 

•4 

cases to be reviewed. (See Attachment 5) 
2. Printout of^liejffc -information is obtaified and given to Case Service 

Evaluators.* * . * 

N • * 

- ' ' * J. Assistant Director- of Field Operation notifies Regional ^Administratqrs 

' - r * and District Adndhistrators, of date of review-, and the Administrator^ 

of Evaluation provides them with'd-list pf capes 'to b$> reviewed. », 
'. ' , • . * ' - ( 

( . • C. When Case Keview. is completed one c^pg of the Case ^Review* Sheet J.s given po 

the District Administrator, one: copy, i% sent to. .the Administrator* of 

^ « Evaluation and the original is kept by the ^Regional Case Servic^ y Evaluator^ 

, - (See Attachment 6; ^ * ; 1 rf 

D. , Case Service Evaluator prepares *a sul7nnary of the evaluatiqn findings in * . 

each district. . (Se.e Att^c^ment 7) ' * • / * ^ 

* ■? ' • f •« » » 

IV. Utilization of Case Review s V 

(" Conference.. pi District Office and Regional Office Personnel. 

< - „2.- Allowing at leest ' three 'weeks for 1 District. Office staff to review . 
• - • *\ . • # . ^ ■ 

' - * * tf 

evaluation 'findings the Regional Administrator establishes a date for 

. '+ . \ 

v a conference with District Office persprinel including the District \ 

* ^Administrator, Assistant District Administrator and District 0f9xce J . 

1 a . ' Supervisor?. ' Regional Office personnel are the Regional Administrator, 

Assistant' Regional" Administrator and Regional Case Service Evalijator^. 

. ,2. Evaluation' fipctincfs are renewed and discussed - strong and weak arehs 

' identified, problem case-service patterns^ and areas pinpointed, .need «\ 

^ , for "clarification or modification of procedures. studied, etc. 

* N . 3. District Office t gives its evaluation. Jf ^the evaluation results, and 

* * * prfesert.ts the procedure <5r method it infen&s to u^e to disseminate 

, ' /* . * aiid utii4ze ^valuation results -"and to 'implement corrective action' where 



. necessary'* 



•4. Procedures to be used for follow-up .are discussed. 
Report of Case Review and Evaluation Conference by Regional Office., 
Following the Case Review and Evaluation Conference the Case Service 
h Evaluator and Regional Administrator submit a report on the conference to ' 

» . 1 1 * * "Hi 

the Administrator of Evaluation, which report includes: (See Attachment 8) 

* * / 

* V - & • 

1 . Jden^JLri cation of proWemqi in and recommendations, for soiutij^is to suqj^ 

probl&ps in case service areas, 

The expressed opinions of tahe Distript Office' regarding the Case Review. 



i 



»£n outline of the District Office 's^pians for dissemination and 




tytilizatiop of the Evaluator's finding^ to supervisors *and counselors, 
^ and plans for implementation of corrective action where deemed necelssary, 

\ £ ' ^ " - . 

Report on* Administrator of Ev&luati8n. . 

'* \ * - - ' , % 

Following the receipt by the of f ice of the Administrator of Evaluation of 

ail completed ^valuation review forms', District office Summary Evaluations 

. • ' -o • " * r 

and reports of -District 'Office Case JEtraluation Conferences the procedure is 

as follows:- * 4 m \ * • * 

v * , - * * * ' y 

1. The material from the case renews is compiled, reviewed and evaluated. 
(See Attachment 9) m ^ , 

the Administrator of Evaluation, after analyzing the case review data, 
drafts a report* of conclusions and recommendations. mv • 
3. The draft *is prese m t^d to various Central Office management levels 

' (Director, Assistant Director, Planning Section, Field Operations^ and 



the Training Section) for review, and iri, 
To obtain a user's perspective seveb&l' District, Administrators asked for % 
input on both content and format of the dr&ft. 

' ■ > 

The final product, written Jby the Admires trator of Evaluation, is . 
distributed to the Direbtor, Assistant Director, Planning tect ion, - 
Field Operations, Training Section, .Regional Administrators and 
'District' Administrators. (See Attachment 10) 



!n^ut. 



4. 
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Dm Iinp2ejj*9ntation of Changes. / 

2» - The- Admin^s trator of Evaluation and staff meet with Assistant* Director 

- . of the agency and appropriate central office staff to discuss imple-?, 

. . . mentation of recommendations based on the final report. Final decisions 

v • , \ ■ ' . ■ ' ^ • • » •• r " •*' 

4 on acceptance or rejection of recommendations and implementation 9 of / 

* % % ' % changes* ctr<e made -by the Assistant -Director* - - -- 

• * V * ~ '' ♦ * ' - • * 

•» 2. The staff of the Administrator of Evaluation and the Regional Case 

* ; ^Service , Ei^Iua^ors ar&^vailable as resource 'peopl 6 tq those invol ved 

, V' in- the /implementation of changes. " - ' . 

V. n FoIIov-up 4 / . -1 , ' 1 • • . * 

\ t Following completion of the implementation procedures, the Regional C$se * • • ' 
. ; Service EvZluators', as directed by the Administrator of Evaluation, spot « 
check 'to determine' if directed changes in fact, being implemented.. 

To date t in Pennsylvania^ we have, been utilizing our Case Review 'Process fopsr 
approximately- three years., have gained meaningful experiences ^nd added another* 
fitim&nsiop* toi our 'eyalua tiop prograiti. The fact that our experience with Case HmqLeir 
Jjas.hee^ positive is due, , we t feel , in a- large measure toi careful planning and 
preparation, Sfcra^npd $&r9onne£&nd the commitment of top management -people . Far** 
a/iy evaluation program to h^ye real meani-ngr and va^ue a strong <X)imfitment ,by top- 

management is an absolute necessity * we^are fortunate in PGtitisylvan±a to h$.ve" 

• ^ v ^ - / , * ; 

* * j * " * 

* ■ • ■ * - ** • 

this. JVo -I^ss important is ac adequate*, and trairied* staff*b.f case* reviewers which; 

not only insures that the work will % be done correctly but establishes credibility. 

*> \ ' ' ' ^ # •* 

with* the "field staff, another important factor in. the success ,oj? ^ny .'evaluation 

program. -The four ftL& time case \service reviewers were 'trained- for approximately; 

.three months before assuming, the^. responsibilities of theif new positions. The 



of the 



carefuX planting-' and preparation tliat is necessary includes t£e preparation 
field staff to accept the 'evaluation prodess* / 4 \ * A 

As Eric Hoffqr states % in his book,. The Ordeal of Change , (1963) , "Even ±zt 



slight things the new is rarely without some stiri^Mg of foreboding." : ah agencies 
•at one time or another, have had some type of case, reviews. However, if an agency 
develops and implements a structured^ and permanent, case review process as part of 
policy there may be some "stirrings of • foreboding ". To prevent this we suggest an 
• ' awareness/ type of training at all' levels prior to the implementation of't&e^case' 

review jyrao*ss_. ^The^urpose.-of-this~training^ould-be to- present the'how, why and - 

»where of the case- review system. If personnel are aware of the purpos.es and 
procedures. Aere wp^e^ minimum of stirring.^ In -Pennsylvania we produced a video 

< tape of the Casei Review Process througrfbur Training' Section. This v^deo' tape^ was ' 
•shown in each o^our 15 Districts Office's. Following thj> showing of the tape a team 
of BVR personnel nkde up of individuals from Case Service Section, Evaluation 

Section, and Training Section were available for qaestions and comments .' In attend- 

f> ^ 1 ' ' ' / 

•• ane ^ at cl i3ese ™ etin 9s »ere all personnel in the District Office, and our experience 

with this training, and the outcome were very positive". 
^ * The Case Service Review Process, like program evaluation' in general, is 

'neither simple nor easy, but injpennsylvania we have found it wellworth the effort. 
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ATTACHMENT 1 

COMMONWEALTH OF PENNSYLVANIA 

June 27, 1980 • 



subject: Case Review 



to: 



District 'Administrators 

Attn: Placement Counselors 



from: -Joseph P. Weir 
Administrator 

"Systems Development Section — 
Per : KendajDjJj i /jfcfeng 

Plac ( enfenfT US^cjinator 

J 



A Case Review on selection p'f vocational goals, and assigned D.O.T. codes^ 
by the Evaluation Section and the^Case Service Evaluators will begia 
July 1, 1980, Placement Counselors will be asked to participate in ih±& 
.review by determining job" availability in their area. The evaluator 
will complete a Log of Vocational Go^l Selection and Job Availability «*\ 
(see attached) to include all cases reviewed for each district. The * j 
. evaluator will complete columns i, 2 and 3- during the case review. 
Placement Counselors will be asked to complete item 4 "Job Availability 
In Area" with a Yes or No response. Item 4 should be completed by Placer* 
■ment Counselors using tile ^cjjertise and knowledge of the local labor maxfttfb 
and will not require individual case file review. " . ' 

.Should you havg- questions regarding participation in this review, please 
'feel free to contact pie. Thank you fgr your usual cooperation. 

L * , 

cc: Mr. Latz # * t 

Mr. Kennedy 
Mr. Williams 
Mr. Guise , 
*Mr.* Brandt 

Regional Administrators 
Evaluators « 
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COMMONWeACT^ OF PENNSYLVANIA 

June 9/ < 



. SUOjfcCTi 



TO: 



Case Review 



i?e^l^nal Administrators 
-District Administrators 



Case Service Evaluatoz 



\Adolph Lat z 

Assistant Direct 




/ 



L ih! ! 7" seIect2 ° n of national ?oa2 and assigned DOT codes 
_ gf-^^^gg-Jggfaon an4 j*e_Cage.Jteriei, ^valuator, will Wn ' 
• on July 1 1980, since the District Office Placement Counselors have 
expressed concerns About the correctness of the DOT code that is 

ofllTLTJ^ !f OC ^ iona2 S° ai ™* *>e appropriateness of the selection 
of the vocational objective. •; /, . 

The correctness of the above information is. necessary to assist .» 
counselors m the client vocational selection process. 

^1 WB WU1 dete ™*™< on a Statewide sampling - 

basis, the preyalencij of these concerns. . 9 

" ?o%Zl? tS %e C&Se Revie *« U1 * ™e* *» the Placement Specialists 

J&rss^-** existin? trainin# ™ s - ™**< ' 

✓ cc: A/r. tfagran 

A/r. Kennedy 
Mr. Williams 
Mr. Guise 

Mr. Weir * ' . 

Afr.- Flemirtgr „ v " , 



> 



2. X« client's worA* history 
\ recorded? • ' 



case res ato wi yss w am 

(26) .1 i 
j 1 



f PAUL G 
YE$ NO NA YE 

7V~* 1 



GEORGE TOTAL 
YES NO NA YES 110 NA 

5 ' 

1 3 1 



2. Xs DOT code correct for 
the stated vocational * 
goal on the BVX-194?- 





(26) 
(28) 



2< 
2 



4 

2 



3. Kas~ extensive vocational (26) 
ccal counselling required? (28) 



2 

1 



If Yes to Question 3, was (26) 

"...*it provided? ' (£ff) 



1 
1 



4 

-JL_2- 



5. ' Vocational goal on BVR-194 
realistic in terms: 



J 



A. Disability and 


(26) 


1 


1 1 




2 






limitations? 


(28) 


1 


1 ' 1 




2 






B. Available job 


(26) 


1 


i I 




2 






opportunities? 


(28) 


1 A 


1.1 




2 




; 


C. Skills? 


(26) 


2 « 


2 i 




2 




T 




(28) ■ 


2 




.2 




2 j 


t 


Has vocational objective 


(26) 


. 2 


2 


2 




2 * 




chanced (BVR-195)? % 


(2ti 


1 


*2 


2 




2 


Z 2 



7. Xs DOT code correct for* (26) 
* the stated vocational goa^ (28) 
on the BVR-19S?-. . 



1 
1 



1 
1 



2 



2 
2 



4r 

4 



8. Was extensive vocational (26) 
ooal counselling required? (28 f 



1 
1 



Xf Yes to Question 8, was (26) 
it' provided. (28) 



1 

1 



*2 
2 



2 

1 



4 
4 



4 
4 



Vocational goal on $VR-295 
realistic iji terms: 



A. Disability and 
0 limitations? 


<26) 
(28) 


Mi 

1 ' 2 j 




4 ' 


B. Available job ^ 
opportunities? * 


' (261 
(28) 


2 -»J 2 
J -2 2 




* 

;4 


C. Skills?? 


(26) ' • 


— ^ — : — * — . v" 






Total 


9 


3 16 10 4 14 9 6 13 8 


7 13 : 36 




Percent of "Yes* 


758 


71if - ^60* '\ 53Z 
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w ' ^AmCHMENT 4 

CASE REVIEW S&EET OF VOCATIONAL QBJECJIVE_ 

Status 



3/30 



N = No 
N/A » No$ Applicable 



CLIENT: 



AGE: 



CASE NUMBER 



D. 0. 



•4 $ _j 


ansTT 


COMMENTS: • 


- «f- — 

1- Is client's work history recorded? . 




— <ir- ' 


2~ Is D.O.T.^code correct for the, stated vocational goal 
on the BVR-194? . > 




*• 


3. Was extensive vocational goal counsel! inq required? ' 




• 


4- If Yes to Question 3, was it provided? 


ft — 




3«. Vocational goal on BVR-194 realistic in terms: 
JL Disabil\ty and limitations? 


41 




B. Available job opportunities? ' % 






Z. Skills? ' ^ -° 






Has vocational objective changed (BVR-195)? 






TL 3s O.Q.T, code correct .for the stated vocational 
uoal oh the BVR-195? 






8* H?s extensive vbcational goaV counselling required? 






9* 3 If Yes to Question 8, -was it*provided? 




J 

m 


10. Vocational goal on BVR-195 realistic 1ft terms: 
A. Disability and limitations? ^ * 






&• Available job .opportunities? 


f — 




C. Skills? \ 







V 



ft. 
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**# t f *" « ATTACHMENT 4 



• . 3/80 



.ft 



* . euiD &HlES'JOR C/jSE REVIEVf OF VOCATIONAL OBJECTIVE 4 

* 3 — 7 m : Z " i » 



REFERENCE: 



\, . - * -1 # Emphasis should be placed upon*the dent's employment ' Rev: RPM 402.02 , 

--history (principal employed, length-of-employment-, : -to- 402^036 — 

' ' and duties and responsibilities). If the client has . BVR 104 Rev, 6/75 

% i no work history, this fact should be recorded or 

indicated. ' > * * ft 

. 1 N % 2. Vocational goals*shoul3|be define* on the IWRP. ' 76-430-20 

Correct DOT should be "serted and rtby'reflect tyo 
digits (occupational grouping* or job family). \ 
* Shou|d an old R-lTtfe the basis for the vocational * _ 

^ _ objectwe, no DOT may be shewn and should, therefore, & 

be considered. not applicable. - ^ 

*p 

3*' This question should be answered YES or NO,, and the . RPM 409 . 
latter is not considered a negative response, § 76-430,20 

^ Extensive 'vocationalfgoaT counseling should be - ' - \ ' 

documented in the selection of the Vocational goal, . * 

Where a prima facie vocational goal is shown such • l» 
as job retention, the judgment of the Evaluator may 
^ /*■ % • be that,.extensive vocational goal counseling is not ^ 

' \ rCqUired * * % 

^ ^ 4.Mf, in the judgment orthe Evalultbr, extensive ,' RPM 404 

; * • n vocational goal counsel in^ho.uld be provided, 76 " 430 ;4° 

/ / 'the case narrative or progress notes should 

t. v T reflect that &ch counseling- occurred. If>io *\ , ( 

$ such recordings are#*n the case file, the <^ \ " . . 

response should be ML If the Evaluatpr finds * * 

v that extensive vocational goal counseling was . . * 
rtot^equired; then the response should be NOT 

J|PPU CABLE. s n ^ ; ' • 

^ *' " : r -r.._ r x. i «k,V,f^m chnnlrl ' • RPM 417 - - 




The Selection of the vocational objective should . * w 4 il 7 on 

be -justified and an explanation of hew and why 76-430.20. 
the client and counselor reached 1j|is decision. 
The vocational objective should coincide with ^ , ^ \ < 
the client's work tolerance, performance ability, * 
skills ^d knowledge, occupational orientation/ ^ < 
attitudes, and understanding of responsibilities. * . 

A. «%The -physical and mental demands ^ % 

compatible with the physical and ^ • \ ^ 
e Rental* limitations of the client ] 
atfd other personal factors. 



• B. Employment 'opportunities should be. y 
% ~ . considered' at tyie time of the selection* " « 
\or the vocational objective. ' . * 
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• . . ' . * - ( 

C. Client has sufficient skills; is adequately 
pxfcfpare'd for the job.- ^ 

Con'finnatitm, revision, or major modification of the. 
original goal is- included in the amendment along , 
with the* rationale for the change. . 



REFERENCE: 



76-430.20 



See Guideline No. 2. 



1 * 



See Guideline No. ?. • _ • 

Counseling and Guidance should be documented in 
.the change of the vocational objective. 

See Guideline No. 4. 

See Guideline No. 5. 
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^ ATTAC&.EN?t4~"- 



QUESTION INTERPRETATION 

FOR *" . 



VOCATIONAL' OBJECTIVE REVIEW SHEET 

U YES WoA.t hUtotej JLa AecoA.ded [took fan job £ctte7 weefeCy (fining*, and pviiod wotted 
* . . ^oa. each, job dieting woAfc -ti^cl, 

NO Job ^t£c,\ccfcfr/ e<&n£ig4 oa. period uwtkerf no£ rioted ^oa. ohe ok molt job*. 
Exptoui uude^' COMMENTS 1 ', 

"*.•*„■ 
MA *- Ct£e*tt hcti ncveA woA.feed, 

2* ' tfES - fl.O.T, Co*. u'coAAcat ^oa. 4*oted l/,0,' ai pea job' and Q.O.T. Code Ziiued at 
euaCuittoA. meeting. Two digit cocfc -c6 A.eqtu*ed, fonn. d&it code -ia A.eqLUA.ed^ fan. 
ZioniewafeeA • ;/ * \ 

.WO^- 0.0. T. Code i& not coAAeat £oa. 4^atedA/,0,, ExplaUn unde^COMIE.\rrS'\ 

31A - .Ccue it> old and hcu> no IWRP, V,0 t i& A.ecoAded on auz najutativo. t<uAhau£V*0<>T/ co 



3- IfcES - Evaluation judgement - tonAidzn total acue corLterttA (coriA-ideA.: cliznt had nev&x vconh 
ptydwlogical touting; on-thi-job tAaJbihxg} othzn. tAabung) cZi&nt'A motianaL ttat 
v . unnzaLiAp} ctiznt 1 * vocational hai\dicap t I . - 0 . 

' T© C££en£<t6 A.ctuA>ung to oa. co>vtuuung vooxk at 4ame job U*e,, cticnt X* get££ng 

^ ttonnout 'aJiti^icifyl Limb Azplac^d), " ^ 

1 iiA r ^ not iue, * • ' 4 * 



4. ViES *- Eva^aioA.* judgement - pAootf t/tat extenA-ive. coun4e£ing tax* pAouided <U doc.uzi2Ti£p.{L 
on exuz naJinktivton. caae pA.ogA.e44 note, ;. ; • \ " 

i/0 . — ' PaooiJ that exteni.cve cotutaeting tea* provided -ca not docaweitted tin caae ncwia&vz on 
ca4e pAogA.ei4 note, Exp&tin undfcA.. "COMMENTS". - , 

NA - ' ttoe urfien question No. 3 i6 aniweAed "NO". . . . 

5A /ES Evaluate*, judgment - Baaed on. documented dUabititiz^ and lim-itcitioni r it appeals 
* t/iai cLcatt 4/touid be ab^to peA^oAw^he stated V,O t competctc.veCy. • - 

f/0 -A EvaCuatoA. jtidfloneitt - Baaed on doewnzntcxi d&ablLLLiu> and Limitation* 9 it appttt& 

• tiiat oJUant witt not be ablttto p&i&om-thz 4-Cated U,0, compot,LtLve.tij, Explain 

• undoA "CaiUEWTS", . > - 



l?o not use, 
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Job* <uc rtvacCdbtc. 

Jobi a,te not avxUabtc. - cUatt mm' /wue to Jtebcafe, etc. 
Vo Not (tic. ' ' 

Uya&iate.t judcjunuit - Baud on tvotik kUto^y Education- "tut teJif** *i • 

competitively. 



wueionat ot/ecUve ccpSSiJ^ au ""-"»* *» !»*«*» ike . 

Po jtat'tue. 

* * 

Vocatccnat objective urn changed a& hicondzd on 8VR-I95. . 
Vocational objective cau not c/zangecf, 1 PO MOT ANSWER QUESTIONS 7.S,9, 'AW- 10.' 
Vo not tue. '„ - , 

«4e«uc4tlon -intzApJietation Vata wdvi question nwnbvi 2. ' 

tfce quz&tion intlApKetation data undeA qutUion-iumbU*-.- - , ' . *' 

tMe quution inteApietatun data undvi quution numbvi 4. ^ • 

CHe question interpretation data under question number 5. 
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Altoona District Of fic« 
*- . TAJiiVsilEBT 

VOgAttoMAti OAtewtv* cA$a teVifcU 

1 33 -Cases ■ * 



jf# 3s client's work history recorded? • 

2. Is D.O.T. t Code correct for the stated §oal' on the 

BVft 194? J . 

^3r~~Was~extHnyive vocational goal counseling required?... 

If v Yes, was it provided?..... 

4. Vocational goal' on BVR-194 realistic in terms of: - 

A. Disability and Limitations* i . . . 

B. Available Job Opportunities . s 

C. Skills , v ... 

5. ' Was vocational objective changed (BVE^195)? 

6«, Is D.O.T. Code correct for 'the stated vocational 

goal on the BVR-195? * 

7. Was extensive vocational goal counseling-requiredi . . r 

If "Yes, was it provided? 

8. Vocational. goal onBVR-J95 realistic in tlerms of; 
A.- Disability and Limitations 

Available Joby)ppo^inities •••• 

C. Skills - 4 
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0.0. 
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COMMONWEALTH OF PENNSYLVANIA 



.Altoona- District Office Summary Evaluation 

of Vocational Objectives f , 

\ - 

Harry -W. Guise,. Administrator 
Evaluation -Section 



October 30, 1980 



K1.0M: William J. Arnold, Administrator 

', Doninn T TT tit' 



r \. 



i 



Region III 
Per:' Paul E,"Saupp, Case Evaluator 




The following summary evaluation provides an explanation of the results. * 
of the recent vocatienal objective review conducted in the Altoona 
District Office. All items will be reviewed where necessary, to 'indicate 
areas of weakness and'strength. The findings are as follows: N * 

ItenU - This item was -one of the weakest in the. review -si nee six of the 

cases'.did not have an adequate work history, although sixteen 
of the applicable cases did^ have one. - The main discrepancy was fa41ure to 
give detailed, employment histories, including -major gaps' without explana- 
tion, as well as omission of length of employment in some cases.'.. Since the 
employment history isthe foundation for proper goal planning, it seems a 
structured history should be re-instituted to facilitate securing complete 
employment histories. • % . - ' , n^iete 



Item 2 



\ This was a very strong area in -that all cases reviewed ^recorded 
• the correct^D.O.T.' code. 



ItemJ, - - Twenty of the thirty-three case's reviewed required extensive vbca- 

■ • fyonwl counseling since the elitents had no employment history, ■ 
were uncertain *of their goal direction, or needed. a new objective because of 
the employment handicap caused by the clients' disabilities and physical 
limitations. The remaining/thirteen, cases didviot require extensive voca- 
tional counseling -since they>planned on returning to previous employment • 
or they, had sufficient work experience, which would have eliminated the need 
for extensive vocational counseling. In" addition, these disabilities and 
limitations were such that* extensive vocational » counsel ing was not necessary. 

■ Item 4 „ " This was -the weakest "item in the review since eleven of the . * 

a ' • tw ^ nty cas . es tn ° a - t required vocational counseling did not receive 
it. ♦ Again, the results point oujt the 'need for emphasis of the provision of 
vocational counseling as ,an important professional aspect of vocational 
rehabilitation. -Considering the amount of funds spent for providing client 
services, as well as the effort expended by^he counselors in the rehabili- 
tative process, it seems that it is incumbent upon ,us to justify the voca- 
tional decisions that- are being made on behalf of qur clients. "Also, when 
S U Ihi de rj. h0 J lm P°rtant the selection of a realistic vocational .objective 
llo Wn* \l m t te ^ ccess ° r Allure of the client -in securing suitable ■ 
employment, it. becomes more apparent that the foundation of success is 



ERIC 



X 



• 233- 



2n. 



*s \ . 

adequate goal planning, including effective •vocational counseling provided 

by the counselor. Only ift.this mannef can we' provide a logical' apd pro- » 

fessional basis for the decisionmaking p.rocess in rehabilitation. 

Item 5a - Generally a strong, 'area since only four cases -of the thirty- 
three reviewed did not indicate that" the vocational goals on \ 
the BVR-194 were realistic in terms of the clients' disabilities and limi- 
tations. All selected vocational objectives were inconipatible with thfe" 
clients 1 limitations and, therefore, the evajuator did not believe the clients 
would be able to perform the work. In some cases, especially those receiving 
physical restoration, the benefit of doubt must be given t the counselor since 
the evaluator was unable to make a judgement as to the improvement to be 
gained in the clients' physical condition through restoration. Overall, the 
selection of vocational objectives seemed to be handled in a satisfactory • 
manner by the counselors. 

IjgP! 5b - Since about a third, or eight cases f of the thirty-three^rerfaied 

did not have available job opportunities to match the selectxorr 
of vocational goals in< the clients' home counties, this seemed to be a reTa^ 
tively weak-area. The lack of available job opportunities could, in thesis, 
cases, be related to a poor' selection of vocational objectives because of" - 
the absence of vocational tounseling in th£se cases. One aspect of this ftenr- 
that was overlooked was the matter of relocation. Since we^pnly considers* 
the counties where the clients resided, a^negative response could have hem 
fisted on the, placement counselor's list of job opportunities'/but , in reality, 
the clients would have relocated in some cases if opportunities were avaxtable. 

Uero 5c - A very good response to this item since in t only one case of twenty 
reviewed the client did not seem to possess the skills necessary to perftrrnr 
the selected vocational objective. In thirteen cases the arrangements were 
made- for the clients to acquire the necessary 0 job skills through training. 

Item 6 - The vocational objectives were changed in three of the thirty-* 
three cases reviewed. 

The comments previously stated in items two through five'would apply' to 'alt 
remaining items in- the review; i.e., items seven. through ten. 

In summary, the Attoona District Of Wee evaluation revealed that work his- 
tories on , the BVR-104 need to be more detailed <*nd complete, vocational ■ 
counseling neejjjs much greater attention since it # was the weakest area, andf 
more emphasis should be placed on selecting vocational ^objectives that have 
greater job opportunities available. Otherwise, the areas reviewed such 
as correct D.O.T. codes, selection orreal-istic vocational goals in respect' 
to disabilities, limitations, and skills were handled ih a very satisfactory 
manner by the .counselors . 

PES/1 md * 
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COMMONWCALTM OF PENNSYLVANIA 



January 19, 1981 



subject, AUoona District Office Vocational Objective 
Case R'eview and Evaluation Conference 



-TO, 



Harry W % Guise, Administrator 
Evaluation Section 



M 

William J. Arnold, Administrator 
Reg.ion III 
Per:' Paul E. Saup'p, Case Evaluator 
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The Altoona District Office vocational, objective Case Review and Evaluation 
^Conference was conducted on January 12, 1981. The participants for the con- 
ference included the following: Messrs. Arnold 'and Saupp from the Regional 
n Office; Mr. Buydos, Altoona District Administrator; Mr. Shae, Altoona 
Assistant District Administrator; and Supervisors Sarson ,' Fet-terman and 
Drass. Supervisor Hoover was absent because of illness and Supervisor 
Greuber had a prior commitment. 

The Altoona .vocational objective review included thirty-three cases. The * 
' evaluator discussed each item on th"§ case review sheet and 'el aborated on the 
items that had the greatest discussion. The itenft that elicited the most 
response were as follows: . . 

' A ■ 

Item 1* - Structured vs. unstructured format for employment history on the 
-j. >BVR-104 was discussed. The consensus among the supervisors was • 
that trie unstructured one would be more acceptable- since t'he counselors should 
use-th6ir own judgement in determining which work history is important in their 
/goal planning and which isn't.- The evaluator replied that all emp-loyment 
* history -could be significant in -belter understanding the client and making 
sound vocational decisions*. Also,, that. counsel ors may overlook -important 
* aspects of employment if the format is not structured and, vocational 'decisions ' 
will be based , on incomplete facts.' * 

J_tem_4 - E^en though tWs Hem had the greatest number of discrepancies sin/el 
twenty cases that needed extensive vocational counseling did not * iii 
have it provided, there was little discussion of this item., Nearly all the 'Mi 
participants agreed that vocational counseling should be provided on C.P.N. iMl 
and/or typewritten narratives to reflect the interaction between counselors™ 
• and clients in jointly developing the I.W.R.P. District Administrator Buydffl, 
indicated that his revised case progress could be instrumental in showing MS 
impact and provision of vocational counseling in arriving at the selection) 
of vocational objectives on/the BVR-194's. ' • 

' Item 5 " Supervisor Sarson stated that even though vocational counselinfe 

v was not evident in eleven of the cases in item four, that t/$fc ' 

.apparently sound vocational decisions were 'being made by the counselors Mfause 
•or the low number of discrepancies in. this item. The evaluator stated im't 
tms seemed true "based on an analysis oT~all information' in the casefileP> 
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however, item four onl-y pertained to the case recording aspects of vocational 
planning. \Di strict Administrator Buydos injected information about 5B referr- 
ing to interpretation of thisMtem by the placement counselor. ' Since eight' 
of the thirty-three cases did not seem to have realistic vocational gcjals * 
based on available job opportunities, it seemed that this was a result of 
the placement counselor only considering/potential in the local labor area 
rather than if only onepositfon existed. This seemed to be at variance 
with the interpretation by the other placement counselors in the region 
since they all^listed fewer discrepancies for this i'tem than the Altoona* office 

Item 6 - The vocational objective was only changed in three of the thirty- 
three cases' that were reviewed in the office. 

The other questions in the review elicited discission similar* to items two 
through five for those cases where the Vocational objective was changed, _ 

At *th$ end of the conference, the evaluator asked about tf\e procedure •for t&e 
'local office, pertaining to' securing client's signature on a BVR-195 whar* 
the counselor finds out that a client has changed^ the vocational objective £ 
of his own volitionf at closure. The present polfcy >n the Altoona District 
Office is to insert a narrative in the^casefile at closure to record thfist t 
information and not secure client 's signature on the BVR-195. The consensus . 
eemed to be that it would be very impractical . to- attempt to. secure, clients:* t 
ignatures since many of them would not sign*jt after securing employment 
and because of expenses ^o^mileage incurred irt making trips Tor this purpose^ 
The evaluator agreed with^this. position as a practical. matter, but statedT f , 
that the* regulations call for securing\the clients 1 signatures on the BVR-I9o r s 
when a change of vocational objective occurs. As a result of this discussion* 
the evaluator stated that clarification of this situation' would be secured , 
from the proper source, „ . 

Messrs. Arnold ,and Sauptp wer§ favorably received' hy the staff and particf^ 
pation of the "group was very satisfactory. Hopefully, the discussion tfiak' 
occurred should lead'to a greater awareness of the^ importance of case record- 
ing in arA>ving at realistic vocational objectives in a logical manner. 
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545 cases 
V TALEY SHEET r \ 

VOCATIONAL OBJECTIVE CASE REVIEW 
STATEWIDE 



1. Is client ! s 'work history recorded? 



2. Is D.O.T. Code correct for the stated goal on the 
BVR 19^?. ..j i 



3.. Was extensive vocational goal counseling required?. 



Tf Yes, wasNit provided?, 



Vocational goal on BVR-194 realistic in terms p£: 
A. Disability and Limitations., » 



Available Job' Opportunities •'•,..«..< 

Skills. .\. 



5. Was vocational objective changed (tfVR^WS)? .\ 



6. Is D'.O.T. CoJe 'correct for Che stated vocational 



goal on the BVJML95? 



i \ O J J 



7. Was expensive vocational goal counseling required?. 

If Yes, was it provided? .T7\ . 

8/ Vocational .goal On BVR-195 realistic in terms of: 

A. Disability and^Limitations . . . ; 

.By Available Job Opportunities ^. ; , 

c. skills 
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■ , Case -Review Objectives 

* * £he Program Evaluation Section and the>Case 

Review Evalup.tors initiated the vocational object- . 

f 4 

ive review on July 1, 1980*, since the District , 
Office Placement Counselors had expressed concerns, 
about the correctness of^ the <DOT code that was 
assigned to the vocational goal and the suitability 
•of the selection oL the vocational objective for 
the client. fc / , 

Through this case review we plan to determine, 
on a state-widetbasis , -the prevalence 7 of these con-^ 
•'cerns.. Also, -it is planned that the results of this 
case review 'would be used by the, Placement Specialists 
to. develop training or modify existing training pro- 
grams, if necessary, for Placement Counselors,.. . • 

You will find in .your reading of this executive 
summary that, on a state-wide basis, the counselors are 

£} • * 

effectively functioning in the selection of a suitable 
vocational objective and in the correctness of t the 
assigned DOT £ode. 
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Case Review C haracteristics \ 
— i t 

' • ■ *» 

The total cases reviewed was o49. ' 6 . 

The cases reviewed were selected on a state-wi.de 
random and stratified sampling basis. 

The 549 cases were made up of Statuses 12, 14, 
* 16, 17, IS, 19, 2Q, 29, and 39. Clients' cases 
- with a vocational 'goal of homemaker.or workshop 

employee were not considered, for the revi#S*»-' 



The "initial vocational goal and any subsequent 
vocational goal change was reviewed for suitability. 

~ * * • 

The initial assigned DOT code' and anx, subsequent ' . 
^DOT codes was reviewed for correctness. 

This review included some procedures that -had not \ 
previously been initiated in any other jreview:' 

a. The Placement Specialists assisted in developing 
the objectives, the measuring instrument, and*' 
the standards for this review. - 

b. The .Placement Counselors in each District Office 
f . assisted by making a judgement on the selection 

■ 'of the vocational objective based' upon the 
availability of job opportunities. 
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Conclusions 



The conclusions will center aroun# two primary . 
questions: - ' 

■ i . . 

1. Was the vocational goal realistic? 

t 

2. Was the DOT code ^correct for the stated 
- , vocational goal? , • . 



These two questions included both the initial voca- 
tional goal on the BY'R-194 and any subsequent vocational - 

• . . . * i • • • 

goal on the BVR-195. . , 

To, assist as in out assessments of the realistic 
selection -of the vocational goal, we considered three 
factors; namely, client disability and limitations, ayaiX— 
able "job. 'opportunities*, and client -skills . 

Selection' of Vocational Objective , , 

1. On a state-wide basis, the initial vocational go&L 
on the BYR^194 was realistic in: / 

. a. 91%~of cases reviewed in' terms. of* disability and 
limitations . ■ 

4 b. ^94% of cases reviewed in terms of availability of 
• job opportunities. ' * « 

c. 87% of ckse£ Reviewed in terms, of client ' slf ills . 

2. On a state-wide basis, the subsequent vocational goal 
-on the BVR-195 was realistic in: 



a. 97% pf cases reviewed in terms of disability and 



limitation. > 
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. b. * 97% of cases ^reviewed in, terms of availability of* 
' • -job opportunities. • 
, c. 88% of cases reviewed in terms of client skills 

/ • • 

Correctness of DOT Code ' ' • 

*? 

1. -On' a statewide basis, the . DOT' codes assigned to 
vocational -go^ls was correct in : 

a. 93% of cases reviewed for the BVR-194. 

b. 90% -o'f ca^es reviewedyfgr the BVR-1||. : 

The above' information -does support the counselor 
effectiveness in the areas of selection of vocational-. ' 
objective and effectiveness of assigned DOT codes'. 
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Recommendations 



The counselors are to be .commended for- their 
realistic selection of the vocational objectives 
,for their clients. - * 



More^ -emphasis should be placed on documentation oi" 
the counseling for vocational goal selection, especially 
for the selection of thk initial vocatipnal objective. 
This is bas4=£hupon the responses to Questions 3 and 7 
on the S>tate-wide Tally^gjiQet . .« 
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545 Cases » 
TALLY SHEET , 
VOCATIOKAL OUJECTTVE CASE REVIEW 
STATEWIDE 



Is client's. work history recorded?. 



Is D.O.Tj Code. correct for the stated goal on the 
BVR 19A? 



Was extensive vocational goffl counseling required?,. 

If Yes i was it provided? 

Vocational goal on BVR>-194 realistic in terms of: 

A. Disability and Limitations , ♦ 

B. Available Job Opportunities 

C. Skills , 



Was vocational objective changed (BVR-195)?. 



•Is. D.O.T. .Code correct for the stated vocational 
goal on the .BVR-195?. ... 



Was* extensive vocational goal counseling required?. 

^f Yes, was it. provided?^. ; 

Vocational'goal on BVR-1>95 realistic in terms of: 

A. Disability and Limitations/. .< 

.D. Available Job Opportunities * 

C. Skills ^ 
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• CASE REVIEW SHEET OF VOCATIONAL OBJECTIVE 
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, Y s Yes 
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N/A » Not Applicable 
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Status: 
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CLIENT: • - AGE": 'CASE NUMBER 




D.vO. . 
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COMMENTS; 
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Is client's work- history recorded? 


— — * — 




.1 « 


C • 


is u.u.u code correct for the stated vocational goal 
on t!\e BVR-194? 


/ 






3. 


Was extensive vocational goal counselling required? 








4. 


t 

If Yes'to Question 3, Was it proj/ided? • , 










Vocational gtfal on BVR-194 realistic in terms: 
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. A. Disability and limitations? 
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Br _ Available job oopdr.tunities? 
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C. Skills? 
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Was vocational objective changed (BVR-195)? 
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7. 


.Is D.O.T. code correct for the stated vocational 
goal on/ the BVR-195? 
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: 8. 


( Was extensive vocational qoal counsel! inq required? 
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9. 


If Yes to Question 8, was it provided? 
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Vocational goai on BVR«:195 realistic in terms': 
A. Disability and, limitations? t 










B. Available job opportunities? * 
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C. Skills? 
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GUIDELINES FOR CASE REVIEW- OF VOCATIONAL" OBJECTIVE 



1. Emphasis should be placed upon the' client's employment 
history (principal employers length of employment, . 
and duties arid responsibilities) . If the client has 
no work history, this fact should be recorded or 
indicated. 



REFERENCE : 

Rev. RPM 402.02 

to 402.036 

BVR 104 Rev. 6/75 



,2. Vocational goals should be defined on the IWRP. 

Cprrect DOT should be inserted and may reflect two 
digits (occupational grouping qt job family). 
Should an old R-ll be the basis for the vocational 
objective,- no DOT may be shown and should, therefore, 
be considered not applicable. 

3. This .question should be answered YES or NO, and the 

latter is not considered a negative response. 
/ , Extensive vocational goal counsel ing* should be 

documented in the seleqtion of the vocational goal. 
Where a prima facie vocational goal is shown such 
k as job retention, the judgment of the Evaluator may 
be that extensive vocational goal counseling is not 
required. 

• 4. IfTin the judgment of the IvaluatorT extensive A 
vocational goal counseling should be provided, 
the case narrative or pro3res%/notes should 
reflect thai such counseling occurred. If no 
such recordings a/*e in the pase file,. the 
respoase'should be NO. . ]tf the Evaluator finds 
that extensive vocational goal counseling was 

4 not required, then the response should be NOT 

I APPLICABLE. . 

5. The selection of the vocational objective should 
be justified and cin explanation of how* and why 
the client and counsel or* reached this decision. 
The vocational objective should coincide with 
m the client's -work tolerance, performance ability, 
# skijls and knowledge, occupational orientation, 
attitudes, x and understanding of responsibilities.^ 

A. The physical and mental demand; are 
, compatible with the physical, and 
mentaT limitations of the client 
and other personal* factors. 

; B. Employment opportunities should be 

considered at ^ba-'fiine'of the selection 
of the vocational .objective. 
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^ C *. Client has sufficient skills,, is adequately 
\ prepared for j^he job. 

6. ' Conn nation, revision, or major modification of the 
* k . . original goal is included in the amendment along 

with the^ rationale for the .change. 

7. See Guide! irn. No. 2. * 

/" 8, See Guideline Ho. 3. * ' 

Counseling and Guidance should be documented in 
the change of the vocational .objective. 

9. r See Guideline TJb.». ^ 
10/ See Guideline No. 5. " 
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Introduc tion , * } 

The plan for dissemination and utilization presented herein represents 
an* expan^ioti ^and revision of the material originally outlined in the Technidal 
Proposal which was submitted* for, this contract. At this point the plan 
preserits both the wbrking model and some of the operational details which x 
appear to be necessary in order to effectively disseminate the findings of this 
project- in such a manner as to have maximuA impact on other* rehabilitation an$ 
related agencies of a similar nature and, structure. At this point it.should be 
clearly recognized that these plans are -tentative and fluid and it iVanticipa- 
i ted that there will be substantial revision to them as the advisory committee and 
coordinating contractor have the opportunity"Tor iriput into the dissemination 
and utilization process. There has been considerable planning .accomplished on 
this process as part of an initial" joint effort by all model states in meetings" 
in Oregon and Washington recently. From these meetings has emerged a general 
consensus and agreement about major parameters of effort and there have been 
preliminary attempts to both initiate and coordinate activities at this early 
stage. One of the areas in which agreement has been achieved is to utilize a 
model for dissemination apd utilization, proposed by Doctor Thomas' Ba.ck6r as 
^part of training to wh^ch all^ states were exposed. That model is detailed In 
the following section and is utilized as an implicit strategy for the develop-^ 
ment of the specific method? and techniques proposed in this paper. 

The dissemination and utilizatioh strategies which are outlined are broken 
into two major area?: internal and extendi activities. The internal activities 

r 

represent an a'ttempt to reach and impact on functions within the immediate bound- 
aries of this agency.- '♦The exte'rnal efforts conceptualize two main, thrusts: (1) a 
sene^s of effort^ conducted individually by .this agency and particularly at other 
"large", rehabilitation agencies which are similar in size and structure and (-2) 
the -other effort is one in which all model states combine and coordinate their 
activities in order to have maximum visibilit/ and influence on a nationwide 



basis. It is anticipated that considerably mor^ planning and development undef 
the direction of the advisory cbmmittee and cooWinating contractor will be 
necessary in the latter instance although it inappropriate to undertake some * 



initial actdvities^|n^lieu of their guidance at this time and a summary deScrip- 



L^W 

tion of several pbjtential activities is included vin this, paper. It- might also be 
nqted that numerous ideas and techniques were developed 4 by the model states in the 

\ 
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aforementioned- meetings and should serve as a valuable" base on Which these 
* fr groups can be^gin work although the details of these are not presented at this 
time/ 

A Dissemination ,and Utilization Model ' „ 

Although Pennsyl^nia^ had originally proposed the use of a dissemination and 
r utTTizatio'n model outlined by the Institute pn Rehabilitation Issues, subsequent 
meetings" with the model states and training of this group in an alternate model 
has led us to adopt the Activities Mqciel proposed by Doctor Thomas Backer in ordei 
to utilize a working strategy ^that is congruent with that adopted by other states 
and so tliat a single model will be employed by alt. The description of this mc&el 
which follows is adaptted directly from Doctor Backer and- certain modifications 
proposed by the State of Oregon. Oregon describes the model- as follows: Figure 1 
is a flow chart based' on the six activity areas given in Doctor Backer's Activitie 
Model.* We have; however, chosen to represent his ictivitiesyfT^through VI as- 
activities forming a natural progression, while' his activity] I - A<|minis trative 

Activities - we have omitted from the^flow chart. This is because administrative 

... . * • - ' > 

i activities, while they are vital to any process, are not conceptualized here as a 

separate set of activities taking place at one specific period of time, but a^ an 

V 

ongoing mainteij^nce process, and' administration ' is assumed to overlay the entire 
process a£ represented in the' flow chart.'. • * 

The process of dissemination ^nd utilization begins in activity 1 - Needs 
Sens^ig activities. Before products or ideas can be disseminated, the needs of 
the potential .or target users should be.identified and defined. Only tljien can 
products (or ideas) be developed to jmeet these needs (BCti\|ity 2). A feedback 
mechanism tp the needs sensing s*tage must be bui^t.into the process, both to enr- * 
sure that product development does not progress based on false assumptiona of . 
user H^ee^ and to ensure that changes in user heeds are takfen into account as 
soon as possible; . ) - ., ^ 

pncg the product(s) has been developed and evaluated bas£d on uger needs, the 
material is disseminated in the most a^ptoprikte manned (activity 3). This can 
range frOm use of the mail service to a full-scale presentation by product Sevelop 
ersJ Mere delivery, however, is insufficient to etisure utilization and, -so', activ 
ities directed at Encouraging the correct use of. the product(s) are undertaken^ * 
1 (activity 4). * , • -/,**. 

These'utilization activities are directed "specifically at th6 product o£ 
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products developed in stage 2. However, to facilitate" a- lasliq/ impact, on the 
agency or unit, skills needed to utilize specific "products Wt be supplemented 
by the acquisition of more generalized skills ana knowledge such that .agency. . 
•personnel can^carry^ut their own evaluative tasks. Hence, developmental as well., 
^as utilization capabilities must be strengthened (activity 5); 'The distinction 
between utilization activities and capacity-building activities can be illustrated 
best by citing the example of the distinction between teaching people to bake' *. 
bread from grain provided^nd teaching people to grow their own" grain. To make ' 
the instruction more than a one-ti*e' effort activities must be undertaken' which - 
seek to increas fi -the capabilities of the users themselves. Once user capability 
is increased, it is. logical^to . expect that user n^eds will change, and hence, the' 
entire process is put infco motion again. - ■• ♦ 

. One area of the model which would appear to ^eed- further amplification is the 
psychological or, internalized dynamic process by which the -uncorporat"it» or/intro- 
jection of the materials or activitie*s takes place =and* is translated into action. 
Stanley Portny conceptualizes this process ihjthe ma>n^represented by "Figure 3. 
He suggestsxthat it is a complex process whiclf i^-media^d by several "variables ' 
and which must proceed through a nudber'df. ^ue*itial stages.' Especially important 
here is the process of internalization" in 'wfl'&h the, potential user/comes to not , 
only understand' the material which is '..being disseminated but develops .belief and 
trust in the ideal under consideration so that *the knowledge -presented in -the dis- 
semination materials can be translated comfortably into- a Series of actions. jThis 
process of internalization allows the potential user-to overcome manyof the ini- 
tial resistances and set-backs which are typically encountered in implementing new 
techniques or prbgrams. Focus on this dynamic process suggest a whole realm of 
psychologically-based strategies which would ^pear to us asf essential if 'effect- 
ive utilization is to occur. • ■>'• 

Objective _ * ' ' < 

The objective of dissemination and utliz'ation plans as required by and as es- 
tablished by the Wntract and the subsequent planning report f is to: 

- Assure the Greatest possible use of' contract products by internal and -* . 
, .external uSeVs * * ' « 

- .Coordinate, to the extent possible, product, dissemination and utilization 

with other contractors ' * 

- Utilize outsidej£esearch~utilizatioh specialists for Consultation and 
training. * * * 

/ - - 

4/ 

• - . L • • *°s : ' ■/«'■-.' 



ACTIVITIES MODEL TCR RESEARCH UTILIZATION PROGRAMS 

Thomas E. Backer, PhD 
Human Interaction Research Institute 

This simply stated model for alresearch utilization program . 
*«is organized under functional activities rather than' objectives^; 
Six main activity areas are postulated i. 



< • . I. 
# 

Administrative 
Activities ' ^ 

This includes pre- 

ration Qf reports 
RSA^ paper work 
concealed with the 1 
host agency or BSA, 
planning, advisory 
cannittce,*etc. 
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ii. 

Needs 
Sensing > 
Activities 

'Itiis includes 
all efforts ^ 
to learn- 
about who th£ 
recipients of 
services a^e/ 
ought to fee, 
and, what; sjpe- - 
cific services 
or information 
they would 

, like to re-' 
ceive.. 



Ill 



£V, 



Product 

D e v e 1 opmen t^^rB i s s emin a t ion 
Activities ~ Ifcctivities , , 

This includes ^ 
assembling, re- ali efforts to ~ 
Tinrng, package convcy i^forma-. 
jng, field ^ tion to poten- ' 
testing, yli- „ Ual users ^ ..' 
dating and 
evaluating in- . 
formational 
materials • 
It also^in- 
cludes evalua- . 
tion^or ac-c: 
coiirttab'ility 
purposes, • 



v; 

Utilization 
Activities 

This include^ 
all efforts 
to promote 
1 adoption or 
adaptation of 
materials dis- 
seminated 
(e.g. , early 
userHlnvolve-- 
ment in III 
and IV). 



VI. < . 

Capacity. 
Building ** 
Activities ? 

This includes work 
with potential users 
to increase their 
ability to acquire, 
adapt and utilize 
new knowlodge inde- 
pendent of ppccifdc 
services of the pro- 
gram. Internally, 
it includes use of 
evaluation findings 
for program improve- 
ment. 
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Internalization of Knowledge 



1 

Activity 


-> 


Disseminatiq/i 


— > 


■ Understanding * 


— > 


Internalization 


— > 


i^tion 
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TASK 7.1 



, Internal Dissemination and Utilization Plan 
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I. Objectives, " 

The pian for internal dissemination and utilization shares the objectives a$ 
outlined in the general description of dissemination # and utilization for. Task 7. 
Mofce specifically though, this sub-task seeks to maximize its impact on the organi- 
zational structure, policy and program development , .field operations and technical 
capacity of the Pennsylvania Bureau of Vocational Rehabilitation through planned 
and concerted dissemination and utilization activities. In this instance the 
focus of effort is on an audience under the aegis of the Project Director as agency 
and an attempt will be made to reach every staff level within this audience. Wher- 

ever possible this plan seeks to involve potential users in the contract work in 

* 

order to secure their input as well as commitment to final product use. 
-II.- Definitions 



ft is useful to define dissemination and utilization separately for the inter- 
nal and external audiences represented in fask 7 in order to emphasize the differ- 
ences io functions which are involved for each audience. These definitions 
describe principal activities on which the sub-task focuses. Within the internal 
plan dissemination is those activities concerned with determination of appropriate 
product users (distribution lists), preparation of transmittal material and the 
actual distribution of the products. The dissemination of project material will 
employ as many, appropriate means of communication for any single product as is 
possible. 

Utilization is defined as user support activities s^ch as presentation at 
ieduled meetings, training programs, special conferences and a system of communi- 
cations fo'r continuous user support. Utilization activities will stress the user's 
internalization of each product to assure awareness, agreement and understanding 
of goals and objectives, confidence in the professional and technical ability of 
the project staff, and the need for* planned, and appropriate implementing action. 

IIT. Evaluation Section /Dissemination and Utilization Policy 

D and U activities are an important^ phase of the evaluation process cycle and 
as such, D and U activities must be considered'fearly in the development of evalua- 

r 

tion .products and continue throughout the process. 

Program evaluation D and U activities! are a primary responsibility of the 
administrator of the evaluation section. J-n fulfilling this responsibility the 
dissemination of all evaluation products and "the implementation of utilization 
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activities related thereto are to be cleared and/or approved by the administrator, 

D and U activities may be delegated to the section D a^d U staff specialist or to 

the evaluation section specialists that are that ar<> assigned specific projects^ 
*■* » 

and ongoing tasks. 

Evaluation section staff who are assigned evaluation projects or have task 
assignments for,, ongoing and recurring products are responsible to the administra- 
tor of the evaluation section for D and U activities that are necessary for the 
efficient and timely use of those products, 

A staff member in- the evaluation section will be assigned as D and U staff 
specialist by the administrator of the evaluation section. The D and U staff 
specialist will be responsible for D and U activities that are generaJ. in nature - 
and specifically assigned by the administrator. The section D and U specialist 
may also function as a coordinator for D and U activities that pertain to specific 
evaluation section products, i . 

All evaluation section personnel should be continuously aware, of the need for 
coordinated D and U activities in order to achieve the highest degree of efficiency 
and effectiveness in field and staff usage of evaluation products. 

For the, purpose of achieving a common understanding that will assist all 
evaluation^sectiffn staff in developing and implementing Dissemination and Utiliza- 
tion plans, Evaluation Section Output is defined in three categories: 

, Recurring evaluation products that have been established and are provided 
to r users on a regulated basis 

V New products established by needs determination and developed through the 
evaluation process cycle. The new product could be furnished on a one-time 
basis or become a recurring product as described above, 

, Informal' evaluation information or *da,ta usually provided verbally or by a 
'brief report within a^ort time period. j 

IV. Dissemination and Utilization Planning Phases 

For the purpose of organization and management, internal dissemination and 
utilization activities are established in four-phases: planning, get ready, 
.implementation and evaluation, Although' there is some overlap of the time period 
for each phase, the activities *in each phase are distinct and must be completed 
before proceeding to the, next phase. In addition to using this type of .sequence 
for the overall project, it should be noted that it could also be applied to each 
of the five steps outlined in Backer's (1979) model since this represents a 
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Organization' for Internal Dissemination and Utilization Organization Chart 
Control and Coordination of Product Producing Tasks 



Distribution 



User Lists and' Dis- 
tribution - 



task Coordinator 
Overall Supervision and 
Coordination with Other Tasks, 
Central Office Staff and Field 
Activities 



I n f o rma t io rv^nd 
Technical Assistance 



Direct User Assistance 

o,r Referral to appropriate 

Staff Specialist 



Task 4 



Provide User Lists, 
Plan and Implement 
Utilization 



Task 5 



Provide User Lists, 
Plan and Implement 
Utilization 
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Task 6 



Provide User Lists, 
Plan and Implement 
Utilization 
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logically organized approach to each pha'se or-activity noted^ in the model. 

The following is a brief ^description of some gf the general activities 
appropriate to each phase^which we plan' to conducts 

1. Planning Phase 

- Selection and consultation with external resources* 

- Development of methodologies and techniques 

- Staff assignments x t 

- Coordination with other tasks and agency, Central Office i 
and field staffs 

2. Get Ready Phase . : < * 

- ' Training 

- Conduct promotional activity 

- Developing* user lists 

-t Establisl> % distribution center and system 

- Establish technical assistance center 

3. Implementation Phase 

« . * 

- Clear and distribute products 

- Determine <and provide appropriate utilizations-support to users 

4. Evaluation Phase 

- Conduct agency evaluation of- dissemination and utilization • 
effectiveness 

- Select and utilize external resources for evaluation of 
dissemination and utilization effectiveness 

V/ Organization, Staff Responsibilities for Internal D and U 
1. Section Organization for D and U 

The structure of fhe organizational design for internal dissemination and 
utilization is shown in Figure 4. In order to optimize the utilization of staff 
skills and time, a specific delineation of the role responsibilities for this 

sub-task has *been made: * *_ 

•Project Manager/Principal Investigator - Product Clearance 

- User support activities 

(utilization) 

- Coordination with agency staff 

- Coordination with other tasks 

- Evaluation activities * 

Task Coordinator(s)/Staf f Specialist Plan and conduct' initial promo- 

designated for D & U activities y tional activity 

Establish and* maintain user lists 
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Task Coordinator(^>/Staff Specialist Establish and maintain distribu- 

disignated fop-*tf& U activities, cont'd. tion system and center 

t * 

, 4 - As directed by the task coordi- 

nators, provide administrative 
assistance 

EDP Specialist/Appropriate Specialist(s) EDE support for user lists 

Provide technical assistance 

Task 4, 5, and 6 task and sub-tdsk 

coordinators For each project, eahfly in the 

planning and development stage, 
prepare an initial plan fdr 'dis- 
semination and utilization (see 
format, Appendix A). Implement 
project coordinated Utilization 
fc plans 

Regional Case' Service Evaluators Assist in user: support activities 

(utilization) 

2. Task Interrelationships 

To assist in organizing and coordinating the manifold efforts required in the 
internal dissemination and utilization activities it is useful to examine the 
relationship* that these activities have to other tasks, particularly the locus of 
responsibility for certain of the planning and coordinating functions between 
tasks. We find that some tasks^(4, 5 and 6) are ones that produce certain speci- 
fied products that will require dissemination and utilization efforts of a similar 
nature while the other tasks have activities or products that will require rather 
disparate efforts. * 1 

A. Product producing tasks: * 
This includes tasks : 

4 - Field test new standards «► * 

4 » 

5 - Field test medical/vocational facilities reporting systfem 

4 

6 - Develop- new evaluation capacity T 

These tasks require certain research activities to be conducted and 
reports presented that are rather similar in. nature. It appears that^ a 
relatively uniform organization and dissemination and utilization effort 
might be effective for these tasks. The responsibilities for the .dissemi- 
nation and utilization activities between task 'Coordinators are illustra- 
ted in Figure 5. This role assignment, clearly places the 'responsibility J 
for specific activities in the hands of the person most familiar with Gr 
knowledgeable about a specific component. For example, the coordinator 
for Task 4 who actually conducts 'the research involved in that task 

^ # 
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. sh^ul^l be most capable of drawing up the details of a dissetninati9n plan ' 

; and in this framework^is assigned that responsibility. 

B. Relatipnship with sub-taskl 2.3, 2 f 4, and 2.5 - Staff 4 ssessment and 

<j" , Tjain^ng. ' # ,/ 

The task and ^ub^task coordinators for tasks 2.3, 2.4, and' 2.5 in consul- 
i . . . f 

tation with the Task 7 coordinator will establish and administer approp- 
riate •staff dissemination and utilization training by selected external*' 
resources . 

C* .Establish and maintain a prefessional library - Task 2.6. 
* Coordinate library program dissemination and tuilization activities Stith 
sub-task 2.6 coordinator. 

D. Staff contact with leacters and specialists in evaluation Task 2.7. " 
In coordination with Task 7.2, External Dissemination and Utilization, 
plan <and utilize to the greatest extent feasible consultations and 

, training to improve project staff knowledge and skills in dissemination 
and utilization methodologies and techniques. 

Consultation And training have been initiated Or are planned with: 
Research Utilization Laboratory for Evaluation at. ICD 

- University of Michigan Research Institute 

- West Virginia Research and Training' Center 

- Edward Glaser Associates 

- Stanley E. Portny, Consultant, Program Management and Evaluation 

E. Administrative Organization and Monitoring*, Plan and Tracking System 
Refinements - Task 3 

n Assure a coordinated dissemination and utilization project effprt for all 
tasks and that those plans and actfivijties are within the project goals 
and objectives. ■ 9 I 

F. Relationship of Task 7 with Task 8 - Evaluation 

Evaluation of dissemination and utilization effectiveness will be con- 
ducted internally and exterrvally. % 

The Task 7 coordinators will . be responsible for internal (agency) 
evaluation activities by designing evlauation instruments, conducting 
evaluations and analyzing results.. Technical assistance will be provided 
by tile Task 4 coordinator. * 1 
. The Task 8 coordinator will be responsible for the evaluation of internal 
dissemination and utilization activities by outside resources. 
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Activity * : ' 

Dissemination 
• .Product Approval - - 

t 

Dissemination plan (See Appendix B) 
Distribution 



Project Managet/ 
Principal Investigator 



Responsi- 
bility > 



4,5, '& 6 



Utilization 

, Approvals and coordination ; 
\^ Utilization plans (See Appendix B) 
Plan implementation 



X 
X 
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The dissemination and utilization efforts will be concentrated around three 
primary activity areas: prpmotion, distribution. aptTvitiei" anff^nf6?mdfiSay^^^ % 
technical assistance. Details of these areas will be presented in the following 
sections. Some individual and group training has occurred-on dissemination and 
Utilization. t,t is hoped these types of activities can continue %o occur for all 
appropriate staff. i • * ' * * - * . 

v The initial objectives for the promotional activities foir the internal' dissemi- 
nation and utilization plan include: 

1. Alert ^potential product users and amplify and calrffy 

a. '"Inyite feedback with eventual input, throughput and output^ ^ * 

2. Promote formal and informal discussion within' the -project and external $0 
the project. 

3. Support need sensing activities* • — * 

a. Establish a base and familiarity for continuing contacts. 

4. Define the time element and impact of project activities. 

a. Suggest the possible products to be supplied by the project. 

5. . Create the potential for assessment of project activities and make re- 
^.^f'^ffements based on needs sensing activities. * 

, The objectives have been achieved and continue- through efforts in several 
areas. There have been presentations made during regular, meetings ^'t the Central, 
Regional antf District Office levels. A number of such presentations have already 
been made. Two types of publications have been used. 6 The Pennsylvania Bureau'of* 
Vocational Rehabilitation publishes "-Success, 1 ! a pewsletter\hat> is distributed to 
all internal staff as well' as many institutions "external 11 to the agency. Also 
"The Rehab Brief" has been utilized and is explained later. ' 

Particularly important have been the development of informal "sharing sessions" 
on a face-to-face level with staff at levels to provide them wittf reformation on 
a variety of project prs^uctg^ well as to gain feedback . on their tuility. 

3. Distribution Activities 

In ccordination with the product task or sub-^ask coordinator, this activity 
is primarily concerned with: 

jtae initial plan for 'dissemination and utilization wili* be* the* responsibility 
of .task c^dinator^ 4, 5 and 6. Effective pre-planning must be undertaken to 
assure *t;he integrity of product use. • 
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- The establishment and classification of a computerized master list of 
potential users for all ."project .products. Primaty dissemination 1 efforts 
are directed to other state agencie^ RSA f and university rehabilitation* 
training programs.. 

- Thf establishment and maintenance of ttie facility that will prepare and 
distribute products to appropriate external users. 

- ,THe initial ^nd continuous promotional activities to inform potential 

'tteer^of project products available and to become available. 
-"This system is the' focal* point of the project as all products will flow. 
* to this point. Because the task, coordinators .are the project manager and 
principal investigator, docuntient clearance and control can be maintaiijed 

. * from this point. . , ^ ' 0 s ^ 

c 

4. Information Center and Technical Assis&nce Activities' 

This activity will providg^iechnical assistance to internal users* related to 
produch: interpretation and is specifically concerned wfth methodologies and tech- 
niques utilized in product Resign, This center will also be both a promotional 
tool and a vehicle for feedback on the .products and assistancef strategies. 

. * Thp technical assistance center will primarily respond to assistance requests 
v ^ . ' 

on products th^t are already developed by the project. Communications from the 

y ' 
.center will be in written, telephone and personal form. Internal linkages have 

o m *» - * * f 

been established and* will be nurtured to continually improve the accessibility to 
and provision of the technical assistance center. Assistance responses to users 
will utilize other project task and,* sub-ta$k ..coordinators and technical specialists 
as appropiriate to the nature of the ^assistance request. 

Functibnal areas in which the technical assistance Center may become involved 
- are: • , 4 

1. * Direction of users to library material, as appropriate. 

2. Service requests for data available. • 

3. t Assists users in the identification of interftai or external sources to 

supply informational needs." »■ * 

9 4. Time and cost studies dr requests when information is not readily 
available • 

5. Development of dissemination and utilization activities regarding informa- 
tion to all types of internal and external sources. 1 

6. Coordinate the collection o*f information from other "sources . * 
7., Utilize feedback* for future assessment or refinement. „ 

For further assistance in developing the above thoughts v we 'refer you' to 
Appendices 'A andjj,!*, in telrras of formatting* for planning product D & U activities. 
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\ ' 

VI. Specific Dissemination and Utilization Activities and' Plans 

1. Evaluation Section General Activities f 

Specif ic> types of internal dissemination and utilization activities are 
presently being used iri the Model Evaluation Unit and the Bureau. A variety of 
scheduled meetings have been used to promote and discuss Model Evaluation Unit 
activities ? Among those are Regional and District Administrators 1 meetings, ^ 
various Central Office staff meetings, M.E.U. meetings and special scheduled 
meetings in certain district offices. Several special conferences have been 
attended with presentations given by M.E.U. staff. ; 

The m!S*E.U^ staff produces several publications each month that aid in^alerting 
M.E.U.' staff, potential users and pt^fiote formal and informal discussion with 
supportive needs sensing. The publications are: 

. Insertion to the "Rehab Brief" - M.E.U. staff distribute the "Brief" and add a 
local flavor by producing an insert which contains Perthsylvania information related 
to the topic o*f the "Brief" ; also information available from our expanding library 
•and other "goodies" 9 and freebies of interest to Bureau staff are included. The 
"Brief" \s- then disseminated to each professional staffer in the Bureau - statewide. 
Requests resulting from "Brief" information have been heavy; however the M.E.U. 
library staffer is doing ^an outstanding job of filling the requests in a timely 
manner. These requests have generated the need to use the Pennsylvania State 
Library as an additional - resource within the State. The National Rehabilitation 
Clearinghouse and Project SHARE have been out-of-state sources utilized to. completer 
requests. 

. Library Listing - A monthly list of new additions to the M^E.U.- library is 
developed and produced to increase. utilization and communication. 

•~ M.E.U. Update - A weekly compilation of M.E.U. activities, upcoming events 
and developments is printed and distributed to each M.E.U. staffer every Friday 
morning. • 

. Training Advisory - This monthly compilation of available training for M.E.U. 
staffers has helped increase awareness of outstanding opportunities. A "How to 
; Complete Requests for Training" packet was planned and developed by a M.E.U. staffer 
to ease access to training Opportunities. * # 

. Monthly Progress Report - This • deliverable is published and distributed monthly 
to all M.E.U. staffers. 

An M.E.U. activity "use^ to paonj.tor events is a set pf two program control charts. 
One chart features a one year look at all deliverables an£ then due dates and other 
related contract requirements. Another chart is used to log training, travejrand 
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special conferences dates and activities of each M.E.U. staffer. Both charts, 

3 feet by 4 feet are accessible to all staff at any. time* - 

Needs sensing ^has been conducted at the Central Office level regarding the 

dissemination and utilization of Evaluation Section monthly and quarterly reports. 

This activity was highly successful and resulted in several needed changes to 

streamline reporting. The activity will moVe eventually to the district level, tx> 

^dfTfermine their needs and develop appropriate ' action plans regarding dissemination 

and utilization of Evaluation Section information. 

M 

A system of boxes for each M.E.U. staffer for incoming mail has been developed 
and implemented. This has led to an orderly dissemination of information in a 
fairly timely fashion, although improvements could be made. 

Several topical areas have been chosen as possibilities for an ICD article. 
M.E.U. staff are now in the process 'of developing these areas to be incorporated- 
in several articles or dne. 

- Suggestions have been made to improve the flow of incoming M.E.U. mail. These 
ideas have ireen takpn under advisement by the ^valuation Administrator (project 
manager 

2 . Task 4^ 

Task 5\ Ear }y stages of pretest are being conducted with probably internal * 
— • activity occurring later. 

3. Task 6 . t 
Task 6 Products 

Case Review Studies (CRS) 
Alcoholic CRS 



? 



D and U Status 



SSDI/SSI CRS 

.Vocational Objective CRS 
^Profile Analysis "Technique (PAT) 
~„ Management Information System (MIS) 



1 * 



Evaluation ,Siection\Organization 
£ol low-Up Studies : * • 



Needs assessment, instrument design, ^ield 
evaluation and implementation,, final report 
and D and U conferences by D* and U staff 
completed. 

Same as above 

Needs assessment and instrument design 
completed at this .point. 

' Product presently in design and development 
stage. t 

One "segment of the MIS the Evaluation Section 
Reporting Systems at the Central Office level 
has undergone needs sensing, revision and 
implementation. -User support is provided as V 
requested. % - 

Needs sensing, .developmental proposals, 
'revisions, and continued development of pro- 
posals with presentations is.'in process. 

Need's sensing completed and outside 
consultation. • \ ^ 
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3. Task 6 

Task 6 Products . * D and U Status 

Case Review Studies (CRS) ' cont'd. ^ 

Casfe Review Process ^Program in place with continuous evaluation 

and revision occurring to improve effective- 
ness* 

Similar Benefits Needs sensing, instrument design and initial 

phases of implementation completed* 

Client Satisfaction Studies Data collected and "analysis planning initiated 

Facilities Data Ba§e ■ Needs sensing and initial instrument design 

• * have been completed- 



New Methods Planning for new products/projects is contin- 

. " uing^fpr the third year.* 



\ 
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FIGURE 1: Flow Chart of Dissemination and Utilization- Activities 
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NEEDS. SENSING, 

a. Identification of, Users* 

b. Identification of Needs 



i 



I 



CAPACITY BUILDING 
a. Work with users 



5 



9 
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PRODUCT DEVELOPMEtff . * 

a. Conceptualization,. 

b. Assembling, writing, packaging, 
designing 

c. Evaluating, field testing 

2 

" I 



DISSEMINATION 




a. kill. 




b. Telephone, in-person 


, contact* 




3r 


< * 



UTILIZATION 

a. National Conference 

b. Technical assistance teams 

c. >0n-site visits 
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Pennsylvania 
Bureau of 
Vocational 
Rehabilitation 



Labor and Industry Building - Seventh and Forster Streets 
Harrisburg. Pennsylvania 17120 9 
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Program Evaluation Section 
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WHAT IS PROGRAM EVALUATION? 

A systematic set of data collection and analysis 
activities undertaken to determine the value of a 
program to aid management, program planning, staff 
training, public jaccountability and promotion. 
Program Evaluation activities make reasonable 
judgements possible about the efforts, effectiveness, 
adequacy, efficiency, and comparative value of 
program options. 

WHO DO ES IT? 

■ £ 

Personnel from the Program Evaluation Section have 
a variety of skills available to assess a situation 
and assist in measuring the achievement of goals .- 
Hie section consists of program and management * 
analysts, statisticians, clerical staff and an 
administrative assistant . • - ' 



WHY DO IT? ; 

Program Evaluation can assist central and district 
office staff in determining strengths and weaknesses - 
in all programs and work activities . Program * 
Evaluation can 'help management determine if ' 
further training is necessary, policies and proce- 
dures need modifications andvif goals and objectives 
are being met. — 



HOT IS. IT POKE? m ~ " 

Program evaluation can be completed in a myriad of 
ways . The basic procedure is to determine what needs 
to be evaluated and what kind of information is 
desired to collect and measure pertinent information, 
data collected, coded, analyzed, <reviewe<| with ths 
"actors" and a report Written, and/or orajty presentee} 
with suitable graphics. An extended fo^w-(lP tf| 
conducted Jo determine the impact of the evaluation ' 
as strengths and weaknesses are analyzed, 




EXAMPLES OF WHAT PROGRAM EVALUATION DOES . 

In Pennsylvania-a variety of methods are available 
to complete a program evaluation, some examples are: 

A. Case Review Process - Provides analysis to manage- 
ment of case' service patterns within the rehabilitatior 
process . 

B. Client follow-up Studies - Provides information^ tQ 
management on how the^ rehabilitation process meets- , 
client needs and Federal and State standards for 
quality. 

C Similar Benefits Information Collection - Allows * 
centralized documentation of "statewide erlortsl to 
utilize similar benefits. 

D. Profile Analysis Technique - Provides data for 
statewide and district analysis of selected rehab- 
ilitation characteristics. ' , 

E. Evaluation Instrument Development - We provide 
the capability to design instruments and technical 
assistance in such development . 

|Examples of Questions or PrAlems P rogram Evaluation 
ICan Help With • ' 

Nee$i an evaluation instrument 
•Need to depict .information graphically 
•Determine how to use' EDP. to get information. * , 
•Need raw data analysis 

•Assistance in formatting goals and objectives 

• Assistance in evaluating programs 

♦Development of quantitative/qualitative measures . 

B9R MORE INFORMATION CONTACT HARRY W. GUISE 
ADMINISTRATOR, OF PROGRAM EVALUATION, BDOfil 1317 
LABOR AND INDUSTRY BUIIDING, HARRISBURG, PA 17120 
TELEPHONE: (717) 787-7312 
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APPENDIX 28 

EalA Live Toad 

Each Morntiig^AtuL 
ThatWiUBeThe 
Worst Thing You'll 
Have To Face All Day 



May 1, 1981 

a- 



Volume XVII, NumWr 1 



2. 
i. 

5.^ 

6. 

7. 
8. 

9. 
10. 

12. 
13. 

14. 

15. 
V 16. 

17. 
18. 



The Operational Tracking System for airline ticketa is May 1 - May 15 ; Creative Journiea 
bid completed May 18-29 ; .develop contract, June 1 ; submit contract, 45 working days 
approval July 31 , August 7-14 send out airline tickets. ^ 

George Conn is the ^ew head of RSA. He was)Wmerly with Paralyzed Veterans of America. 
He' was director of Public Information in RSA during Nixon and Ford days. 
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Natibnal Conference June 30-July 2. *' v 

Sister-States Sept! ! 15-18*. 

Most people have a "career clock"; this tells them to reach a certain point in an organi- 
sation between ages 32 .and 45. 1 

Lis Pan's presentation was stimulating and worthwhile — • certainly nothing to just set on 
the shelf. „ / r 

Zelda has completed the April 30 newsletter article. 

John Muthard was. okay — he* will help us most with his contacts around the U.S. and the t 
impressive demonstration Leah gave him on our T.A.C. , " 

The June 30 MSOA is to be 'done by Bill S. 

9 * * * ** 

Tex-Mex is okay but getting expensive. _ 

May Day May Day— —May Day ' 

The ejirline ticket estimates, at this time, come to just under $6,000. Tammy got the 
estimates on. April 29, 1981. % ' 

Bill reports the meeting w/Games went well. Games felt PAT was a good comparison technique 
It was .felt (hat set-up of statistical variables will be done here, and tested up there. H 
will then give us his SAS- techniques,. '* 

Word is out that Sue is 'doing some modeling for John Wanamaker. For the raw evidence, look 
at* the attached sheet. » « - * 

Does anyone know why a farmer would put a hog house r on the north side of his barn? 

Leah has the Library under control. At the present time, typing of cards and folders is 
about* up-to-date. We had' favorable Comments on our library system from' Liz Pan. She said 
it is one of the nicest she has seen, and that our Vertical Files were very -well organized. 
Good work, Leah, but then we would expect that from you. * 

Art and Andy^and Bill traveled to State College on Thursday , April 30. * 
Don't- forget Mother's Day> MayvlOCh. 
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Eat A Live Toad 
The First Thing 
Each Morning...And 
That Will Be The 
Worst Thing You'll 
Have To Face A 11 Dav! 



Volume XV number 3 



March 24, 1981 

1. !»• nation*! oonterence 1, ,et for June 30 - July 1 „t scottsdele, M. 
1. «out 70 px.pl, toja,4J t^-tft. Me. n.tioneX conference venting to etteni. 

5. *an has S9t « return for CSAVR quest£oxmaice f ^ zil 10 _ April J2> 

She agency for the national conference will be finalized April 12-15 at APGA thing. 
1* t San Antonio had real good Mexican food. 

8. There was a gospel singer convention at the hotel Art and Don stayed- at. 

9. A body in the bushes???? * • 

12. Our- sister-states conference is moved from Way to Sept. 16-17, 1981. . 

Zeida is doing hotter W/24/M; and ve nope she be back real soon. . 

'Jtoi is tentati veiy set for APGA on April 14 and 15. 

14. Our new director is Mr : George Lowe- , 

15. The 6*9 survey* are reeiiy coiainy in-. > 

«.* Art viii be speaking on the Fis'*in Philadelphia on April' 22. 

17. Rehearsals for sister-states will begin this summer. 

18. - DickCohenis retirement begins, oh 3/25/81. Best wishes to a fine person. ' * 

19. Andy allowed the analysts to reyiew his honeymoon and evaluation will be made in next up-date 
TtlS^ZSJSf' * WteJ * ****** «» — * of Aprii on the statu, 

21* Sue , is in love with Harold. A 
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Eat A Live Toad 
The First Thing 
Each Marnirig...And 
That Will Be The 
Worst Thing You'll 
Have To FaceAllDayj 



February 6, 1981 * Volume XIV, Number \ 

1. Our Section staff participated in a Training Seminar on Technical Writing on January. 28, 

29 and 30. We did some technical writing assignments, and feel we learned some new "twists 1 * 
that; we can put \o use in our unit in not constructing long sentences, since nobody likes 
.to read sentences 'that are 'a paragraph long, so we are really trying to remember this as 
we $o, this issue of the Up-Date,, so that our readers do not grow weary of reading, and we 
can condense much more information" into many fewer words as our instructor "toad" U3 that 
short sentences tend to command the attention of the reader. 

2. ^ Susanne Choissier is a Contract Officer fo* the fledgling MIS Contract to be let in March, 

April, May June or July* 

3. We are to use APA for style for future MSOA - this is just for typing and formatting - see 
Leah for specifics." p 
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4. Horace Greeley once said, "Venture forth toward the sunset, male neophyte." (I think..)' 

5. The National MEtJ tape slide show is jaue out March 2, 1981. 

6 ( . Harry will address APGA April 14 in St. Louis and NRA on Sept. 3 in Minneapolis, on our 
Contract. „ * 

7. He may also appear at the CSAVR Spring meeting from April 28 to May 1. 

8. tt is not mandatory that, "A single illustrative graphic contains the value of myriad basic 

thought units." (Moreover this slogan is NOT a Sinogenic homily) f • 

«■ • 

9. A NERO Conference will beheld in Massachusetts and other MEUa will make presentations on 
April 29. 4 

10^ The National Conference is now set for* June 24 & 25 - location to be announced. 

11. MEU Newsletter input is due from us on these dates: April 30, 1981 & July 31, 1981, to ETC. 

12. M§0A stuff is due March 31, 1981,, June 3fr, 1981 and September 30, 1981 to BCTC. 

13». Critical Review of BPA Program Standards Report is now due 60 days' from receipt of Berkeley 
document (projected receipt 4-30-81). This was due January 31, 1981. 

14 Review of BPA Project Standard Report is now due 60 days after receipt (projected receipt 
April 30, 1981). This way due Feb'. 28. 1981. 

15. Review of BPA draft of training materials is sow due 30 days after receipt of BPA report 
(projected receipt June 30, 1981). This was due on Aug. 31, 1981. This may be done on-site 
by BPA - who knows? « 

16. Independent Analysis of FIS r due Feb. 28, 1981. 

17 Review of Walker - N ARF draft receipt expected May 1, 1981 - due 30 days after receipt. 
18. Individual performance standards are due to Harry on February 17, 1981. 
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1*. M«ct Sister-States Conference feting February 24, 1981, UX Bl«^, 

20. How. let's get serious. ™ 

21. Lit Pan has been subcontracted to do a library training program for MEU 'librarian.; she will 
call soon to do needs sensing. 

,22. Do not enumerate your prospective fowl, until the process of incubation has been fully 
developed . 

23. WVRTC was planning to use our Sister-States needs sensing letter for their National Con- 
ference needs sensing. 

• 24. Only overt desire for commercial currency becomes the Riveting genesis^of ^"""ious^ 
malf actions, appearing to fill the bill that "The love of Honey is the Root of all Evil. 



25. That's it. 



6 
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Contributions of Technology to Deaf 
and Hearing-Impaired Individuals 



Imagine a deaf person being able to pick up sounds from a 
hearing aid implanted in the brain. These kinds of tech- 
. niques, as far out as they seem, are already in the develop- 
ment stage and eventually, in some format, they will be 
brought to a marketing or distribution stage,- Dr. Frank 
Withrow, Chief of Educational technology Development ' 
for the U.S. Office of Education, told members participating 
in workshops entitled "Application of Technology to Han- 
dicapped Individuals," held at the Library of Congress in 
November 19)9/ * 

For many years, great strides in electronics have been 
made. Techniques'have brought about developments such 
the miniaturization of equipment and devices and the 
packaging of video and audio tapes in cartridges and 
cassettes to simplify recording and immediate playback. 
, Tneseadvancementrfiave enabled the design, construction, 
aad.production of specific components which contribute to 
devices either especially designed for or adapted to the deaf 
-aad hearing-impaired person. * 

« , 3rea4casf Television 

Most TV programs depend heavily on spoken dialogue. 
Fast cuts, flashbacks, and visual supplements can only con- 
toe a viewer who cannot hearlhe accompanying sound. 
However, two principal techniques have been developed 
and are available/or making the spoken portions visible— 
captioning and sign-language interpretation. Captioning, 
wnich is kdding the dialogue in print on the TV screen, has 
the widesEappeal since interpretation poses many problems. - 
; Forihe broadcaster, such problems arise as: Where should 
the insert be placed? What size should it be? A more-serious 
problem, however, is that approximately three quarters of 
those who cannot hear or understand speech do not under- 
stand sigrr language. 1 « , 

Technical Aspects of Captioning. * 

Captioning technology has been' directed to respond to * 
two alternatives: ' 

*) Captions which are shown as part of a regular broad- 
cast and appear on alt TVJscreens (open captions) or 
only on TV sets equippcdlith special decoders (closed^ 
captions): and . 

J : D ' a J^ 0e,k -> !T ' *** ^ffopuhtUm oftht United State*. 
Silver Spring, MO: National Allocation at the Dtaf. 1974. 



b) Captions which a>e prepared in advance (off-line cap- 
tions) or generated simultaneously with the dialogue 
(reaMme captions). 

Open 'Captioning 

While no technical problems exist in using open captions, 
they are not regularly provided. There are two major rea- 
sons for this: 1) the broadcasters fear that captions will be 
objectionable to hearing viewers, even though there is very 
little research to support this assumption; and 2) writing 
captions is difficult, and using them involves added expense. 
For these reasons, closed captioning has been the favored 
alternative. 

t 

7 _ * , Closed Captioning ^ 

The television picture is constructed of 525 lines, of which 
the first 2 1 lines in each image field are blank and can be used 
for other purposes. Line 21 is being used to carry captions 
for deaf and hearing-impaired viewers^ 

Three television networks, PBS; ABC, and NBC are pro-- 
ducing approximately 20 hours of programming each week 
utilizing this system. PBS (Public Broadcasting Systems) 
began with 4 hours per week and is building up to 12 hours 
each week, a goal planned to be reached by the end of 1980. 
ABC and NBC are planning to do 5 hours each. To make 
this venture feasible, a National Captioning Institute has 
been established on each coast. ^ * 

Real-Time Captioning 

Real-time captioning means delaying, for not more than a 
few seconds, the transfer of any sound or statement in the 
* broadcast into words for viewing. To accomplish this feat, 
computerized shorthand is used. The problem with this 
technique is that shows for hearing audiences are delayed a* 
few seconds also. However, the advantage of this type of 
captioning is that live broadcasts, such as sports, can be 
viewed without extensive delay. 

Off -Line Captioning " \ 

At present, all TV captioning for hearing disabled audien- 
ces is done in advance of a broadcast. Off-line captioning is 
therefore limited to taped progrims. Since most people 
listen much faster than they read, spoken portions are usu- 
ally simplified for the printed versions. Also, because of the 
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language problems associated with hearing problems, col- 
loquial expressions and complex sentence structures «are 
ji voided. 

/Such script editing demands considerable talent and sub- 
stantial time. PBS estimates spending 30 to 40 hours of 
personnel time to caption a* 1-hour broadcast. ' * * 

Teletext 

Current efforts in the adaptation of teletext systems in the 
UrS. are being carried on by the Columbia Broadcasting , ' 
System(CBS). Since their announcement not to take part in 
the joint closed-captioriing effort which began in January 
1930, CBS has been committed to researching the capabili- 
ties of teletext' as an alternate vehicle for captioning 
programs. 

Teletext refers to systems also known as Ceefax and 
ORACLE, which have been designed for the British Broad- 
casting Corporation and Independent Television, respec- 
tively. Since these systems offer a wide variety of informa- 
tion via TV, they can make information of national and 
local interest available on a 24-hour basis. Teletext has the" 1 
added advantage of allowing the transmission of computer- 
generated visuals and interactive materials for viewer 
participation. 

A technical test of teletext began in March 1979, through 
CBSaffiliatoKNOX-TV ih St. Louis. It will run for 1 year. 
According to CBS officials, the purpose of this testing is to 
establish the technical and practical limits of the system 
which is to be^adapteti from a 625-line scan (the standard 
used in the United Kingdom) to the American 525-line scan 
per broadcast frame. > 

CBS has announced their intentions to make teletext 
available for use by Spring 1982. They have also, expressed 
confidence that teletext systems will be compatible With the 
equipment used at the National Captioning Institute. 

Home Television 

Recent developments In home TV systems have dramat- 
ized the potential of this medium for deaf and hearing- - 
impaired people. 

Until as late as 1976, home video equipment was thought 
of as an expensive luxury which required a mass of compli- 
n cated equipment and a sizable amount of technical-knowl- 
edge. Systems, commonly designed for industrial us*, were 
b'tlky and dverly sophisticated. Understanding that a market 
was waiting to be tapped, a number of producers began to 
experiment with basically off-air recording, systems. T^iey 
finally settled on apparatus which functions with small- 
format videotape. The result has been affordable home tele- 
vision with many utilization possibilities for those with hear- 
ing disabilities. r 

. v • 

Equipment 

Equipment targeted for home TV falls intcMwo broad 
categories. The principal distinction is made on the basis of ' 
what carries the program: videotape vs. videodisc. < 

Vldeofifce equipment allows recording of broadcast pro- 
gram directly from VHF/UHF antennae and can operate 
wither without a monuorTh the actuated pyirtg pTOCessvThe 



\ What is Radio TTJf or RTTY? 

I 

An innovative communications*^! called RTTY, ■ 
or Radio-Teletype, was originally used to provide a 
general news service to deaf people in the Philadelphia 
area. The device receives different information simul- 
taneously using only a teletypewriter and a special RF \ 
tuner. The RTTY service to the community has been 
mejded into English courses for hearing-impaired 
children at the Penn-sylvania Schpol for the Deaf. 

RTTY begins with word information that is typed 
out on a special teletypewriter. News is placed on a 
sp€cially<oded^pu*ehelftape. The' completed tapes 
are then fedahrtfuglLa device that changes the punched 
holes in the tape to audible tones. These signals are 
then transmitted by telephone lines to a nearby radio . 
station where they are mixed wi\h a high frequency 
sub-carrier capable of reaching an area within a thirty 
to fifty mile radius. By turning on a specially-tuned ( k 
home radio receiver located near a teletype machine, 
- the listener's TTY responds to the signals it receives, 
accurately reproducing in words the messages pre- 
pared in the studio. ~ i 



equipment does not allow editing of materials gathered a]h'd J 
has limited extra-spanning capabilities, such as slow motion 
Ind |till-framing. Some systems enable users to record their 
own video and audio onto videotape. Most systems consist * 
of a record-and-playback device (videodeck), a monitor for * 
playback, a power adapter, and a timing device to activate 
recording when set. 

Home video systems using videotape generally record an 
electronic image andaudio track onto a cassette. They use 
methods which make the recording process so easy that 
technical expertise is virtually unnecessary. Either broad- 
cast or closed-circuit signals are fed into the machine by ' 
using simple controls. Because of its design, the video- 
cassette is immune to fingerprints, tangling, and improper 
threading. These systems could be used with a caption de- 
coder in order to record programs that would playback with 
• captions. 

The videodisc makes use of a pliable plastic disc which is 
played back on a receiver that uses a type of electronic 
scanner to deliver an image to the home monitor. 

This relatively new* device offers many possibilities for 
deaf and hearings m paired viewers since it contains most of 
the components essential for a'n effective, computer-basecl; ! 
interactive television system: The recording electronics 
allow, closed-captioning to-be recorded on the^isc/anti* 
retrieved through the use of a decoder. A single disc may be * 
'scanned an infinite number of times and rapid selection can 
be made at an extremely fast rate. The speed of the video 
image can be controlled and still-framing is easily achieved 
for extended periods of time. With the aid of a computer, the 
videodisc player can recall and display sequences in any 
order with repetition. » 

The videodisc system is also designed to be easy to operate 
in its playback mode. Programs presently available for this 
system are prerecorded and come packaged in thin plastic 
rcco f d sr-Damagc to the surface of the plastic disc should 
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have little or no effect on the playback 'quality. Mainte- 
nance, 'therefore, is relatively trivial. > t 

7*!c*>iicnic Assistance Devices 

v ^ * *- 

When Alexander Graham Bell invented the telephone, he 
.was seeking ways to aid deaf people. Ironically, his inven- 
tion g ready increased their handicap. The telephone grew * 
Irom an exciting novelty into a virtual necessity, especially* % 
in business and industry. Since deaf and hearing-impaired * 
"jfcople cannot use the telephone*^ originally conceived, - 
they found themselves ex,cluded-ffom many employment 
' opportunities. The social pattern which resulted from tele? 
phone USe adversely affected this group, ami for nearly 100 
years, nothing was done to improve this imposed condition. 
Then in 1964, Weilbrecht. a deaf physicist, invented an 
electronic device that allowed the connection of two telety- 
pewriters {TTY) by means of acoustic/ inductive coupling to 
the standard telephone. x 

Telecommunications Devices for the Deaf (TDD) * 

The acoustic coupler translates electrical impulses into 
audible signals and vice versa. By placing the telephone 
handset in the coupler, two TDD's are joined and what is 
"typed on t)ne machine is reproduced on the other. Since 
neither speech nor hearing plays a role in the communica- 
tion, the acoustic coupler enables sf>eech-im paired persons <• 
as/well as deaf and hearing^impaired persons to overcome 
these barriers. However, this telephonic assistance, device . 
. <Joes require reading and r at a vety low level, ty pin jfa bill ty. 

TDI • ' &*>i 

Telecom™ unicatiojis for the Deaf #> Inc. (TDI), a ■ 
rrofit corporation,, was 'farmed in 1968 to" acquire 



t pe writer costs between $250 an<\ $500. Elaborate installa- 
tions can cost much more, however, depending upon acces- 
sories such as special signaling devices which cause lights to 
flash when the telephone rings, automatic answering devi- 
ces, and prerecorded message equipment such as punched- 
paper tape readers. * . 

In addition to these reconditioned mechanical teletype- 
writers, several relatively new electronic devices are on the 
market and are generally referred to as TDGEsYAQrtually all 
are designed to utilize modern electronic acomp one nts and 
manufacturing techniques which have already been tho* 
roughly tested in electric and computer terminals. These 
new devices typically feature built-in aco us decouplers and 
are therefore partially or fully portable. Message displays 
vary in size and configuration (depending upon the brand of 
equipment) from typed copy on paper to electronically gen- 
erated letters which appear br^a television screen. Not unex- 
pectedly, the new devices are considerably more expensive 
(about Sido to $1,000) than , reconditioned mechanical 
equipment , 



TTY/ TDD 



^on-,' . 

profit a&Vpo/ation^ was farmed in 19£8 to" acquire and . 
recondition teletypewriter^ which majors earners discarded 
as surplus; These reconditioned teletypewriters are then sold, 
at cost to deaf people^ Complete within appropriate acous- 
tic coupler^ a typtcaj}installat|bn of a reconditioned teiety*-' 

J ***iz**==ze^^ to Clswsif^ a Hearing Disability =■ 

^ M uch com usionevojvesfromdistingu^shing'bet ween 
terms such as deaf and hearing impaired; Following 




Vocational rehabilitation counselors and other service 
providers should anticipate a variety t>f initial difficulties 
which their hearing disabled clients may have regarding the 
use of a TTY/ TDD. As with any other service, careful 
instructions may be required before a client is comfortable 
v in acquiring and using a TTY/TDD. g For example, people 
who do not have TTYtyTDD's typically express uncer- 
tainty about their ability. to use one. This uncertainty may 
* stem-frdrn an inability to, type or from limited language 
' skills. -A client's concern about a. lack of typing skills can 
usually be overcome with an increased familiarity with the 
equipment.' The second problem, that of reduced language . 
skills, is more difficult to'overcome* However, with* patience 
on the part of the involved professional and practice by the 
client; a rudimentary level of communication can usually be 
established. 



are some general definitions to help identify these j 
differences. 

Hearing Impaired: A term generally used to des- 
cribe and encompass all types of hearing delects. 

Hard*of*Hearing: A condition where the sense of 
hearing is defective but functional for ordfnary life 
purposes (with or without help of a hearing aid)i $ 

Pea// Deafness: Speech is not usually understood 
even with amplification. 

Congenital Deafness: The ge neral term use d f o r a ny 
^factor causing a person to be born deaf. 

AdventitioltslJeafness: Deafness after birth. t 

Prellnguai Deafness: When deafness occurs before 
the acquisition of language (usually before 3 years of 
age). Such a person will have no language frame of 
reference when learning to speak, write, or speech read. 

Postlingual Deafness: When deafness occurs after , 
the acquisition of language (usually after 4-5 years of 

r~ lifiiinffn ■niri M Tw r»nifnm i n uaeom**** m ■■■■■ 



ge). In^ifiqst cases, persons who have lost their hear- 



ing "after ttus/rfge have a relatively, strong language 
base. 0 - ' 



m ° Conductive Hearing Loss (Impairment): Sound 
waves cannot be transmitted thfpughThe outer ear due 
to a blockage'of the auditory canaUr total absence of 
' the passageway (congenital atresia) or through the 
o middle-ear due tp otitis media (an 'accumulation of 
fluid which blbcks the eustachian tube), usually a te m- 
porary condition. 

Sensorineural or Nerve Deafness/ Hearing Impair* 
ment:,T£c -inner tar or the auditory nerve leading to 
- the brain is affected. It can arise from a variety of 
reasons, including tHe mo'thef having German measles 
* (rubella) during her first 3 months of pregnancy ^com- 
plications of infectious disease, such as meningitis, 
" scarlet fever*, measles, chicken (tax. mumps, or diph- 
theriaVor frorri a blow (trauma).-or toxicity . s 
Presbycusiar deafness or impaired hearing' due to * 



the agirjg jjrocess. 
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. \ Additional Costs 

A final consideration is the cost of maintaining a TTY or a 
TDD long after its purchase. For example, there are tele- 
phone installation charges^ a client does not already havca 
telephone) and charges for monthly telephone usage. Also, a 
TTY ora TDD must be servi^cf/rom time to time. The TD1 
Directory carries a list of service addresses. 

, Equipment for Special Needs ■ 

Equipment is available which isnespecially designed for 
communication assistance. This equipment can permit 
directlong-disunce communication between deaf people or 
between deaf and hearing people. The devices may also be 
designed to transmit information rather than provide two- 
way communication. Some of the equipment described 1 
below is available from manufacturers; other items can be 
custom fabricated. " ' 

Manufactured items available: . 

• Timers that flash a light signal or activate a mattress 
/Vibrator to awaken a deaf person; > ' 

% • Visible M doorbeirthatactivitaies*aflashinglightor,for > 
the deaf/blind person, a' fan 1 , v tf 

• Telephone bells supplemented'with flashing tights; 

• Closed circuit TV systems forcommunicating messages 
or serving as a visual intercom in high security areas; 

• Sound amplifying systems^including hardwire, induc- 
tion loojvand radio frequency systems; 

• Individual vibrating pagers for contacting the deaf per- 
son out of reach of other communication systems; and 

• Message center light systems between receptionist's 
desk and individual rooms,.«pded to indicate phone ' 
calls, messages, or visitors. > 

Custom-fabricated devices: 
. • Three-way switch system wired to door bell that allows 
the visitor at the door to blink a light within the room * 
whether the light is on or off at the time of the signal; 

• Emergency call buttons located conveniently arounfc a 
facility^ hat activate, a central alarm; and 

• Lighted panel that flashes to confirm the emergency 
i message hasbeen received. 

* I * Emergency Devices 

Audio signal systems^must be supple mented with visual or 
other sensory information in order to alert those with hear- 
ing loss. Systems that .meet this performance standard - 
include; ' 1 9 . • 

• Fire alarm and smoke detector system that activate 
strobe lights, vibrators, and/or variable-velocity fans; 

• Fire alarm and other emergency reporting devices that 
do not require voice communication; 

• Graphic messages confirming receipt of emergency sig- '/ 
nils, such as a sign in an elevator cab flashing -Jielp is ori 

the way- when a malfunction is reported: * " 

• Pungent ode-rant adekdjo natural gas supplies, particu- 
larly in sciencelaboratores, t&cu&dea'f persons unable 
to heat* the hiss of escaping gas; -and * » # 

• Warning lights thatjlash on when machines arc run- 
ning, or that signal when a machine has completed its 

k . task. a • 



Visual Ear: A Canadian Telephone , 
Device for Deaf Persons • 

Visual Ear is a low-cost, portable unit which allows 
hearing or speech-impaired persons to send and 
receive messages over normal telephone lines. The urttt 
measures 7 by S*A inches and comes with a carrying 
ca^se. Visual Ear incorporates a small keyboard, LED 
(Light Emitting Diode) display area and an acoustic 
co upier.that enables the user to communicate by typed 
messages over the regular telephone network. The 
main feature of this device is the acoustic coupler 
which accepts all types 6f telephone handsets in gen- 
eral use in North America. Visual Ear is also compati- 
ble with TTY 's and other similar, but more expensive, 
products now On the market. The unit also has a 
long-life, rechargeable battery and AC adapter. 

Major telephone companies across Canada offer the 
product at cost— $350 to persons presenting a medical 
certificate stating the/are hearing or speech-impaired. 
Otherwise, it sells for $375. 



Fgr Wore Information 

f » • 

To contact the National Captioning Institute. Inc., write 5203 Leesburg 
^ Pike, Suite 1500, Falls Cnuich, VA 22041. Phone: 301-99J-2400. 

A> brochure from the National Technical Institute for the Deaf (NTID) 
provides information on commercially available products that perform 
signaling functions. For a copy write NTID, Dept. of Audiology. 
Rochester Institute of Technology, One Lomb Memorial Drive 
Rochester. NY 14623. \ % ' 

A reference book, Telephone Accessories You Can Build by J.H. Guilder, 
unavailable for those who wish to build theif^own signaling items from 
Hayden Book Co., Inc.. Rochelie Park, NY. 

For information on TDl membership and telecommunication matters, 
contact Telecommunications for the Deaf, Inc., SM Thayer Avenue 
Silver Spring, MD 20910. . * 

-A Directory of Services for the Deaf in the United States 1, is annually 
compfed and published by the American Annals of the Otafznd is a 
comprehensive enumeration of educational, clinical, and organiza- 
tional services. For information, write 5034 Wisconsin Avenue, N'W. 
Washington. D.C 20016. 

The Alexahder^Graham Pell Association for the Deaf, Inc. 1537.35th 
Street, tif, Washington, D.C 20007, disseminates information and 
promotes speech and Hp reading. 

A monograph which records the state of technology for telecommunication 
devices for the deaf is Impact 1980 by Drs, Jerome'D. Schein and 
Ronald N. Hamilton. For a copy write the National Association of the 
Deaf, 114 Thayer Avenue, Silver Spring. MD 20910. 

The American Speech and Hearing Association; 9030 Old 'Georgetown 
Road, Bcthcsda, MD 20014, is composed of specialists in communica- 
• tiottd borders. 

The Fall 1978/ Winter 1979 issue o{$he Science Education /Vr** (published 
oy the American Association for the Advancement of Science) deals 
with education and the physically disabled and olfers some valuable 
Information on deafness and hearing impairments. For a copy, write to 
A A AS, Office of Science Education. 1776 Massachusetts Avenue. NW, 
Washington, D.C/200J6. 



Copies oft he BR ft Fa re available from Ur*RRI/Requc« braille or I 
jape editions frbm you* regional service library. I 
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assistive devices, etc. 

The following information i. available fro. the Program Evaluation Section Library: ■ 

,. "Roles of- Counseling in Ensuring the Rights of the Handicapped" - The »™fflB gl 

W Guidance Journal, Vol. 58, Ho. .4, December 1979. 
2. '.Trends of Interpreter Education/Training Programs-Insight, from One- - The • 

. Deaf aVmericap- , June 197?. * 
V "Interviewing Guide, for Specific Disabilities: Hearing Impairments" - U.S, 
• Dept. of Labor*> 

"The Labor' Forte. Participating Handicapped Women, ^g^^g*^ 
* ^he Journal of Applied Rehab ilitation Counseling, Vol. U, Ho. Z, burner 

■ 5 "Facilitating'career Guidance of Deaf Students: Challenges and Opportunities • 

5 * for SunKs" - The Vocational Cu tanea -Quarterly, June 1979.. 
> ' 6 . Acting the Role of .'Counselor" ~. The Personnel and Guidance Journal , Sep.tem- 
ber 1980. , » % . 

7 . - -Mental Health and' Deaf Persona" - .The -Education Dipest." March 1979. 

8. "Deafness: Co^on ^understandings^ - ^2 *»™ al ° £ """"fr ' W ' l91 *' 

- - ^^^^^^^^^^^^ - : 

Journal, December. 1979. " 
,0. "Genetic and Hearing toss: An Overview for Professionals" ^ e rican Rehabili 
cation , Hovember/December 1978. 



."Can You Hear Met Implant B?paa^ Ear's Defect." ■> Science Dipeat. June 1979 
12. "Deaf People Communicatory Phone'* - American Rehabilitation, September/ - ^ 
October 1977. 



/ 



The following publication! are available on loan from the Program Evaluation Section 
Library: 

1. "Impact 1980: -Telecommunications and Deafness" - Deafness Research and Training 
1 Center, Mew York University School of Education, 1980. ^ 

2. "Identifying and Solving Problems: A System Approach" - Roger Kaufman. 

• * 

3. "Hearing Impaired Development ally Disabled Persons: ' A Challenge to the 
Helping Professions" - Larry G. Stewart, Ed.D. . N - ■ 

*********** 

TO OBTAIN INFORMATION OFFERED IN THIS "REHAB BRIEF" PLEASE CONTACT LEAH KUHNS IN 
THE PROGRAM EVALUATION SECTION LIBRARY. TELEPHONE I 8-447-7312. . ^ 

* * * * *•* * * * a 

FREE B f S YOU CAN WRITE FOR ' ^ 

A newsletter is being published jointly by the United Cerebral Palsy Association, 
the National Society for Crippled Children and the 'Epilepsy Foundation of America. 
It is available without cost to interested individuals wanting to keep current 
with legislation affecting disabled individuals. To be placed on- the mailing list 
write: United Cerebral Palsy Association, Suite 141, Chester Author Building, 
425 I Street?, Northeast, Washington,. DC 20201. (Ixmovator"~Vol # 5, No. *3, 1980) 

Regulations spelling out the required implementation of Section 504 have been 
issued. This is t,he fection of the. Rehabilitation Act of 1973 that protects the 
rights of handicapped people to participate fully in community life., These regu- 
lations and the rest of the legislation are available free by writing: tl.S. Dept. 
of Health and Welfare^ 330 Independence Ave., Southwest, Room 5400, Washington, 
DC 20201. People who want to file a complaint in connection with this Section 
may write to the same address. 
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"How to Get Started in Professional Public .Speaking" - Though written for aspiring 
professional speechmakers, this booklet is useful to anyone wholes to speak in 
public. ^It discusses the importance of choosing an interesting subject that you,*, 
know a lot about and delivering your message in a veil-organized manner. 22 pages. 
Order from: International Platform Association, 2564 Berkshire Road, Cleveland 
Heights, Ohio 44106. Free with stamped, self-addressed, business-size envelope. 
("Changing Times" - November 1980) 

"A Woman's Guide to Social Security" - Certain aspects of the -Social Security 
system are of particular interest to women. For' instance, a wife who does not 
work outside the home is entitled to Social Security h^fneWts regardless of age 
if her husband becomes disabled or retires and she cares for a child under^ 18 or 
disabled child who is entitled to benefits. But if the wife does not care for a, 
child, she must be 62 or qlder to receive. benefits if her husband becomes disabled 
or retires.- Those situations are discussed alof| with' other topics, such as* pay- 
ments to widows and employed women who' become disabled, and benefits available to, 
household workers. 12 £ages. Order from: Consumer Information Center, Dept. 
S39H, Pueblo, Colorado 8K309. Free. ("Changing Times" -November 1980). 



.436 



281 



L 



Program • 

» 

valuation in 




oca 






APPENDIX 30 . -. 



OBSERVATIONS 



ft 



U;S. DEPARTMENT OF EDUCATION 

Office of Special Education and Rehabilitative Services 

Rehabilitation Services Administration' 



43? 



282. 



CASE REVIEW PROCESS IN PROGRAM EVALUATION 

HARRY W. GUISE, ADMINISTRATOR OF EVALUATION 
PENNSYLVANfA~%JREAU"OF~VOCATIONAL" REHABILITATION 



^ We in Pennsylvania feel that in^evelopihg a comprehensive program 
evaluation plan, a case review process should begone program methodology 
ift evaluati9p,,activities>* A case review system ha& the capability of iden- 
tifying base linfe -case service' patterns, thereby generating information 
for the initiation of ot impq^veraent of decision making in policy and' 
procedure, - ; k * 



r. The extent and degree of the structure and activities of the case 
review process is contingent upon the personnel allocated by the state to 
evaluation and the 'needs of^the state as s,een by top ^management and other 
inputs, " • ' ; ' 

Because, of the proposed Fe.deral Evaluation Standards which are current 
ly being field tested by the Model Evaluation Unit states each state in the 
cduntry >fill have to develop its own case review process or utilize an 
existing process developed 'by someone else* 

To detine program evaluation ora state level, it is very useful po 
identify procedures fot utilization of information to improve decision 
making^in vocational rehabilitation program planning, monitoring and re- 
vision, (Handbook of Program* Evaluation Studies, Michigan Rehabilitation 
Research, 1978) ThiS* review of current state rehabilitation agency program 
evaluation studies further states that program evaluation studies can be 
divided into three broad areas: (a) input studies which provide informa- 
tion for use in program planning; (b) process itudies which- provide in- 
formation fot use in program monitoring; and* (c)- outcome studies which 
provide, information f ot use in program revision or change. In this 
article our attention will be devoted to the process studies in program 
evaluation studies, r* , 

A process study such as a case reviefr system is concerned with the 
% case 'service patterns within- the rehabilitation process. Examples of the 
type of information gathered from a case review system tM^ht be: (a) 
delayed^ movement of clients 1 cases in the rehabilitation process; (b) per- 
centages of eligibility of served clients; and (c) suitability of selection 
of clients 1 ^ocation^l objectives. This type of compiled data would . 
provide the afJptpptlaxe people in the agency's caSe .service, policy and 
planning sections a "nitty-gritty^ infjormatioA base for change, modif icar- 
tion, development and implementation of policy and procedure, - # * 

For example, we can determine through statistical data the existing- 
time frames in the fnwement of* clients 1 cases, from referred status to 
eligibility status. * If we develop a standard of three -months as an accept- 
able .time frame for that movement we might learn' that in 20 percent of our 
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experience with Case Review has been positive is due, we feel, in a large 
■measure to careful planning and preparation, trained personnel and the 
commitment of top management people. For any evaluation program 'to have 
real meaning anci value, a strong comraijynent by t6p management is an 
absolute necessity — we are fortunate in^fei^asylvania to have this. No less 
important is an adequate and trained staff of case reviewers which not only 
insures that the work will be done correctly but established credibility 
with the field staff, another important factor in the success of any evalur-, 
ation program, - The four full- time case service reviewers were trained for 
approximately three months before assuming the responsibilities of their 
new positions. The careful planning and preparation that is necessary in- 
cludes the preparation of the field s*aff to accept the evaluation* process*. 

As Eric Hoffer states in his^book, The Ordeal of Change (1963), "Even 
in slight things the nfew is rarely without some stirring of foreboding . tr 
All agencies, at one time or another, have had pome type of cage reviews. 
However, if an agency develops and implements a structured and permanent - 
case review ^process as part of 'policy there may be some "stirrings of fore- 
boding*" To prevent this we suggest an awareness type of training at all 
levels prior to the implementation of ^ the casei review process. The purp6se 
of this training would be to present the "how, why and where" of the case 
review system. If personnel are aware of the purposes and procedure^ there 
will be a minimum of stirring. In Pennsylvania we produced a video tape' of 
the Case Review Process through our Training Section. This video tape was 
shown in each of o,yr 15 District- Of f ices. Following the -showing of the' 
tape a team of BVR personnel made up of individuals from Case Service Sec- 
tion, Evaluation Section, and Training Section were available for questions 
and comments. In attendance at these meetings were all personnel in the 
District Office, and our experience with this training and the outcome were 
very positive. v 

,9 * 
The Case Service Review Process, like program Evaluation in ggyeral, 
is neither simple nor easy, but in Pennsylvania we have found it well 
worth the effort. • * 

f 
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EVALUATION SECTION LIBRARY , 

Leah.Kuhns, Administrative Assistant 
Pennsylvania Bureau, of Vocational Rehabilitation 

<n . . / 

The Pennsylvania Bureau of"vocatio^Rehabilitation (BVR) estab-" " 
-lished its Evaluation Section Library as the result of its^Federal contract 
to develop "A Comprehensive State VR Program and Policy System Through a ^ 
Model Evaluation/Management ^formation Support Unit." This contract man- 
dates that a library -dealing with- program evaluatiqn be established. 

' In the beginning.,all that was« envisioned by Pennsylvania's Model Evalu- 
ation Unit (MEU) was!a, small library dealing solely vcfth evaluation .subject 
matter-. But as. the work progressed it became apparent that there was,,a 
need for- a more extensive library which would serve the entire Bureau, and 
the library that was established, while known as the. Evaluation Section 
Library, is, in fact, a BVR library. In addition to its. program evaluation/ 
(PE) holdings it contains material in many. areas of vocational rehabilita-/ 
tion (VR), including new rehabilitation trends, current legislation, specJ/tic 
disabilities, assistive devices, material for civic groups, etc.. This 
library is now housed in the Central Office of -the Pennsylvania BVR and 
serves BVR's 15 District Offices, four Regional Offices,. the Hiram G. 
Andrews Center, (a 500 client rehabilitation center operated by the BVR? 
and'three Disability Determination Offices. 

» » 

- The creation and subsequent development of 'the Library was not an 
easy task. It required careful planning, much research and a lot of hard 
work .on the part ofr the staff of . the BVR EvaluatiotTS^tion. But we feel 
the end result was well worth- the effort, and we would like to share with 
you the process by which the Library was established and to describe it.s 
operation now. , * 

. Since this was a totally new undertaking* in an unknown .area, there 
was an obvious need for expert advice and assistance. This was sought 
f£ JLnV and" VaHST l^cesT" One" of-ttor mcnre^vident neede-wa^r-^- 
Ixpertise of professionals in established Library systems. The Pennsyl- 
vania State Library, located in Harrisburg, was solicited for technical 
SsStance on the establishment, maintenance and continuing, development 
of a Ubrary! In addition, . its staff provided infection on the various 
' activities and* resources. within the ,State Library whmwould enable us 
to supnly the BVR field offices jlth pertinent material to meet theft 
heeS" For example, 'they, provi/ed us. with a list of ^1 P^icals re- 
ceived by the Pennsylvania State Library and placed us on their Bailing, 
list to receive notification of. all 'new acquisitions. In addition, the 
. E^aLatioHection Technical -Assistance Center (TAC)--the official name 
or the Evaluation Section Library-has been listed in the ^-nsylvania 
* State Library's -publication "Directory of Libraries serving the Govern 
men of the Commonwealth of Pennsylvania." ■ This directory shows the . 
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list has been established and books are loaned on a first-come basis. 
Accompanying' each book is a memorandum stating this publication is being 
furnished on a loan basis and the due date for it to be returned. 

Within the next year a separate listing of all periodical^ and books^ 
within the Evaluation Section Library will be prepared for dissemination, 

i 

We have attempted , to sumarize for you how we in P^pnsylvania estab- 
lished the Evaluation Section Library, and how it now operates. Should 
you begin a project such as this, we feel the ma^or emphasis must be oir*" 
taking steps which would encourage user involvement. This is extremely 
important in order to obtain maximum utilization. / ^ 

\ * • / 

fhere has been a great deal of satisfaction derived from the estab- 
lishment of the Evaluation Section Library, especially when we. see the 
interest generated from the field staff. However, we-feel it is important 
to stress the amount of time and effort -needed to produce these results. 
It is by no' means a small undertaking. In additipn to maintaining- the 
actual day-to-day operation of l the Evaluation Section Libary, which in-, 
eludes ordering publications, cataloguing material, and all other aspects 
of library work, we received approximately 150 requests from the field 
-staff during the first year of operation. The 150 requests included 
information offered 'in the "Rehat Brief Insert" and requests from district 
offices and central office for specific technical assistance. Examples of 
technical assistance included information for presentations to civic 
groups on affirmative action for the handicapped, job readiness programs 
for the deaf, current legislation/ arid specific information on disabili- 
ties and assistive devices. We do not wish to discourage you from such 
an undertaking, in fact we wish to dp just' th^ opposite — but we 1 do want to 
make you aware of *the , effort involved. 

We hope thisi information wy.!^ assist you. If you have any questions, 
or want more information, pleale contact Leah Kuhns at 717-787-7312. The' 
mailing address is Bureau of Vocational Rehabilitation, 1318 Labor and 
-Industry Building-,- 7th and Forster Streets, Harrisburg, PA 17120. 

- * . • A - . . 
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Communications with the IBM 3032 that supports it would 
allow inquiries to be made more quickly than our current batch 
systerrt allows. Furthermore, special applications programs, 
could be storedon diskette and easily run against data subsets 



fey relatively naive users. Communications with NARIC and 
J its supporting computer on the East Coast wilr*allow users 
of -the' agency library to access NARICs collection and to 
contribute to the Center's holdings. 



V 

Model Evaluation in an Agency for the Blind 
Wendell Cqx 

Principal Investigator 
Mississippi Vocational Rehabilitation for the Blind, 



Mississippi Vocational Rehabilitation for the Blind is the 
sole blind agency to have received a contract to develop a 
Model Evaluation/Managenl^Unit. While the Agency has 
a strong track'record in rehabilitation and has conducted 
program evaluation for the past several years, this contract 
makes possible the actualization of a personnel team which 
can devote 100% of its time to developing a much 'needed 
and desired management information system. ^ 

In the development of the Unit's Conceptual Framework 
(work plan), it was necessary to carefully scrutinize each phase 
of the Agency's operationlo make decisions regarding projects 
which would have immediate and long-range benefits for all 
levels of management as well as for the Agency as a whole. 
Initially, the Unit made "laundry lists" - list's of who did 
fwhat, when, an4,why; lists of information requirements from 
external and internal sources; lists of available data; lists of 
reference sources; lists -of consultants, universities, insti- 
tutions geared toward evaluation; lists of goals and objectives; 
and, lists of forms, both federal and state. 'All this activity 
culminated in one realization - the need for a strong and 
reliable data base upon which to build a management infor- 
mation system. 

In accord with RSA goals as cited in the RFP for this 

contract the MVRB Evduation/Managdment Unit seeks ±o 

meet the information needs of this Agency through the 
implementation of three projects wh|ch have as their overall 
Objective "determining cosf •effectiveness." 

-An initial project is currently underway to measure the 
population of potential clients. A questionnaire is bein^ 
developed , for surveying the 93 opthalmologists in the 1979 
Mississippi Directory of Physicians. This project also. includes 
forming a working relationship with a local university which-, 
iks computer capability to review the census Jigurefc. The 



. The Model Evaluatjdn 
. William W.J 




ultimate objective of this project is to provide a data base from 
which 'statewide projections can be made, thereby providing 
management with a sound basis for determining human and 
, financial allocations? 

The second study is being desigdfed to yield a profile of 
clients served. Descriptive components include biographical 
data, education, economic status, work history, major and 
secondary disabilities, typ^s of training/services provided, time 
span in program by category, and client outcome. This infor- 
mation will make possible an assessment of the needs of the 
client population. 

The culminating phase of these interrelated studies will be, 
a cost "Benefit study. A cost analysis, including indirect as well 
as direct expenses, will be conducted to produce the financial 
input. The variables to which the costs will be applied are all 
' measurable benefits accrued by the clients from the services 
extended by the Agency/ r . 

V- 

In response to a contractual mandate, the Unit has de- 
veloped an External Dissemination of Information System in 

; order to share project results with other model units amhvith- 
other blimj and general agencies who may be experiencing 
similar problems or involved in similar studies. Input from 
Blind agencies is especially welcomed as the results of otfr 

— worlcwfll probably be -most appropriate for them. # - ^ 

When boVh expert personnel and the funds to develop an 
Evaluation/Management Unit are provided, there are countless 
* areas trtat invite probing and researching. However, each com- 
ponent of MVRB's operation can best benefit from a concen- 
trated effort to develop, install, and make operative a sound 
management information system. The potential benefits will 
, be reafized t not only during the contract per iod, but du ring 
future years of service to the blind as well. 



Project in Pennsylvania 
enkint, Ed.D. 



Principal Investigator * r 
Pennsylvania Bureau of Vocational Rehabilitation 



The opportunities presented by the initial Federal request 
to respond to proposals for the development of a Model 
Evaluation and Management Information System seemed 
intriguing. Closer reading of the RFP, however, suggested 
that the successful grantees would be involved in activities 
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which range far beyond those involved in typical State evalu- 
ation efforts. Even though an Evaluation Section had been in 
existence in the Pennsylvania Bureau of Vocational Rehabil- 
t'ation for several years, we were intrigued by the opportunity 
to develop a'climate in which innovative concepts and tech- 
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niqucs to advance the current state-of-the-art in evaluation 
could be developed and tested. In addition, there was a clear 
mandate to interact with leaders in evaluation from across 
the country. This would expose us to many new ideas and 
stimulate us to explore directions which otherwise might 
remain dormant in the Agency. Many of our ideas, still in 
embryonic form, could be nurtured in such a setting. The 
mandate also posed the opportunity to conduct the type of 
fundamental review of the information and evaluation needs 
of the Agency* that seldom occurs in an organization unless . 
it is in 'the throes of an organizational reorientation. More 
detailed perusal of the RFP led to considerable hesitancy as 
we began to realize the extent of the responsibilities which 
would be involved in this contract. Yet the opportunities for 
impact upon national policy and the state-of-the-art in evalu- 
ation seemed too exciting to dismiss. 

Most organizational theories note that the introduction of 
change into an origouig system can be exceedingly difficult. 
There tends to be an intrinsic resistance to such efforts as the 
organization seeks to maintain its current state of equilibrium. 
Most program evaiuaWs would probably agree that this 
process is made even more difficult by the very nature of 
evaluation activities and the psychological resistances inherent 
in most individuals when they are asked to make their work 
more visible and to increase their accountability. Fortunately, 
was possible to implement the model project in a setting in ^ 
which evaluation had existed for somejime, so that much of 
the initial resistance to evaluation had been worked through. ^ 
The model project was added as a separate unit which would 
interface *with the regular ProgrartfEvaluatiomSectian within 
the Agency. Since the contract required that the Agency 
Director be Proiect Director as well , there is a clear source 
of authority which gave the project the tools necessary to 
implement its specific activities. The basic research direction 
and supervision is provided by the Principal Investigator The 
project manager,, who is also. Administrator of the regular 
Evaluation Unit, provides personnel and ongoing coordination 
between the project, and regular unit, and the Agency.' 

Since the successful: development of evaluation within the 
Agency had occurred on an evolutionary rather than a revo- 
lutionary basis, it was decided that the same sort of approach 
would be taken to the implementation of the Model Unit. 
Thus, following the original publicity and statement of project 
objectives, the primary work of the unit involved conceptual 
and f <f>hiloSophical development with only' slowly increasing 
contact with the field. and administrative staff, so that the 
identity 'of the unit could be introduced in a series of'in- 
1 creasingly visible steps. It seemed to us at this early stage in 
the project's development that the initial task was to begin 
to involve both staff and management in the project during 
its preluninafy phases so that they were .kept informed 
of the general developments and emergence of philosophy 
and approach. * 

One of the most difficult tasks with which We were faced, 
in these early days was the development of a conceptual 
framework. The RFP required that a conceptual framework 
be articulated which is - the contractor's thoughts expressed 
§ -oncepts, of what is the framework or system of activity 



of the State Agency program* and an "analysis of the function 
6f evaluation in the Agency." This conceptual framework 
really represents both an integration of theory and pragmatic K 
approaches to evaluation within a coherent structure that is 
the basis for Building, refining and implementing a Compre- 
hensive informational and decision-making framework. This 
type of macro-level analysis was an activity with which most 
of us had only scant familiarity and has undergone consider- 
able refinement as we-gained experience with it. The develop- 
ment of this conceptual framework requires that policy, - 
goals and objectives be made explicit at both the Agency and 
Section level. We have encountered some difficulties, since we 
do not have an MBO system. At trys point we are in the process 
of developing specific objectives for each of the units. Such 
an undertaking is, of course, one of considerable magnitude. 

One of the first things which we have discovered from thi£ 
project is that it takes considerably more time and manpower 
to accomplish the kinds of activities (especially the derailed 
planning and reporting) which this contract requires. We find 
ourselves constantly having to draw'on the resources of other « ? 
staff to meet these requirements. An allied jxroblem is that 
several of the contracts which were toJ)e awarded at the. 
Federal level for a coordinating committee, n Medical and 
Vocational Facilities Reporting System, and an Evaluation^ 
of the Federal Standards, all of which were ^interface 
directly with the project states, have ;iot been awarded on « 
time. Though this has caused additional work for each of*the 
model states, it also relieves some of the burdens of the actual 
fielcf testing, which would have' been inherent in these coH-^ 
tracts, thereby freeing manpower for other aspects of -the^ 
initial developmental work. ' % 

One of the major Jhrusts within the project is .to develop 
studies of immediate relevance and applicability for the State 
Agency within topical .research areas. At this time planning 
is underway for a seYfes of ^tudies which focus on various 
atopics. within field operations and management information 
areasr Studies anticipated are: 1) the refinement of a case 
review process; 2) a , modified profile analysis technique; 
3) the utilization of similar benefits; and 4)* follow-up empha- 
sizing client perceptions'and satisfaction. Jris 'expected that, 
other areas will be selected for investigation which may involve 
the field testing of a variety of n.ew methods and techniques. 
To insure-that the results of the various studies receive ^the 
widest possible dissemination and maximum utilization both 
within the Agency and at a national level^a master dissemi- 
nation and utilization plan has been developed. 

Currently we are unSertaking an assessment of staff skills 
in order to determine the training necessary to bring staff to 
the highest level of technical competency feasible. To support 
staff development, our professional library, bibliographic 
, resources and technical equipment are being increased. 

An important by-product of the project is that staff morale 
has been greatly enhanced by the chance to meet leaders in 
rehabilitation and evaluation, and the opportunity to develop 
input which may influence national rehabilitation policy 



IV. MEUs - FORMATION AND OPERA f ION 



1 , A PROJECT MANAGER'S VIEW" , 

by Harry W. Guise 

This is a brief written attempt to present a capsule view of 
the many facets of MEU activities from a Project Manager's 
view. This view is not limited to nor all inchisive of all my ex- 
tt periences, thoughts oj con elusions. * 

rn-oombined roles as an MEU Project Manager and an Ad- 
ministrator of an Evaluation Section, I function as a liaison 
person, a reviewer, an advisor, a coordinator, a budgeter and 
an all around Tix-it-person? In these roles, it sometimes seems 
that I spend at least v 65% of my rime on M£U activities and 
55% of my time on my ongoing,activities and responsibilities 
as Administrator! 

My gamut of activities range from the "nitty-gritty" of the 
Building Evaluation Capacity Projects to the Conceptual 
Framewoik of the Evaluation Unit within our Agency. ' 



" "The first definite and very basic thought 
•s the necessity for. commitment and 
cooperation of top agency management" 



j * However, there are advantages to being exposed to these ex- 
periences. Qjie iithat a few deilriite and distinct thoughts have 
come jnjto focus. These ar* not, by far, profound maximums 
but are issues to contemplate pribr to entering into an extensive 
undertaking such as tfce MEU contract. .„ ' ! 

\" Tt. V f *s ? • * 

1 . The first definite; and very basic, thought js the necessity 
forcomraitmeiU and cooperation of top agencymanagement.l. 
I 2. Along with this, you-must have the cooperation of agency 
^ personnel outside your section to assist in developing and ' 
initiating necessary MEU activities. Examples of other, 
agency personnel are Case Service Section, Facility Section, 
. Co^gutft Section and Procedure and Planning Section. ' 

In my role as an MEU Project Manager and Administrator 
, of Evaluation, I have several advantages that assist in carry- 
ing out the necessary MEU activities. In the first place, I have 
access to management and other section h$ads. Secondly, I , 
„ have access to management meetings, regional meetings and 
district meetings. These meetings can be utilized for aware- 
ness training. 

■3. If you are a large MEU state, you must periodically pri- 
oritize your M^EU activities. 9 > 



4„ In a large agency, it is necessarjrperioHitfally to reinforce 
your organizational structure, particularly in regard to spec 
iflc individual task oriented activities.' 

5. In a large state, it is necessary to keep your staff informed 
and aware of the "happenings" through methods, sudh as 

% meetingfrornewsbrieTs. 

6. Toward tHe end of the first contract year, we became 
aware of the necessity to develop and plan for dissemination 
and utilization. We have now developed a "mind-set* 1 for auto- 
matic built in dissemination and utilization for all products. 

7. Early in the contract, it became apparent that Building 
Evaluation Capacity, Task 6, presented opportunities for 
states to develop evaluation techniques and methodologies 
that would meet their needs and result in useful information 
for planned internal and external dissemination, ^ 

8. There must be 4 a basic understanding and agreement of 
contract activities by all participants, states, RSA,and con- 
tractors. If not, the results are: 

a. Loss of effective utilization of staff. 

b. Credibility, within an agency, becomes an issue. 

c. Negative attitudes are developed wfthin an agency. v 

9. Be prepared to make some difficult interna} agency deci- 
sions. If you have no problems with the figst eight issues, 
your difficulty will be minimal. > 



PENNSYLVANIA BUREAU OF VOCATIONAL 

_. _REHAMmATlON_E\^^ 

• LIBRARY t 

Prepared by 
e " i Leah Kuhns , " ^ 

Administrative Assistant 

The establishment of the Evaluation Section Library origin 
nateo 1 as the result of Pennsylvania becoming an MEU contract 
state. 

After considerable research to determine the most suitable 
cataloguing 'system, appropriate sources for periodicals and 
other publications, etc.,4he overall Library concept was formed. 

Since its inception in 1979 the Evaluation Section Library 
has obtained several hundred holdings and is currently receiving 
75 periodicals for use by all &VR personnel. Holdings were re- 
ceived through the purchase of publications and through the 
generosity of BVR staff who donated portions of their personal 
library to the Evaluation* Section for inclusion in the Library. 



The vehicle used to disseminate library information to the 
BVR staff, which includes the Central Office^ 15 district offices, 
4 regional offices^ the Disability Determination Sections, and 



. . The Evaluation Section library 
provides technical assistance to staff on 
specific areas . . . (of) need" 



fourth eva!uator> but current state hiring policies may make it 
impossible to add the position. The new personnef have become 
oriented to the agency and its data, and they are conducting a 
variety of projects withjnereased independence., 

A data processing terminal has been leased andis linked to 
the University of Michigan's computing center. It provides 
access to a wide variety of statistical programs and should allow 
the unit to retrieve and work with agency data in a far more 



the Hiram G. Andrews Center, is the *Rehab Brief which is 
prepared by the University of Florida's Rehabilitation Research 
institute. Included with the "Rehab Brief," a semi-monthly 

c 

publication, is an insert prepared by the Pennsylvania MEU 
which lists available library holdings pertaining to the subject 
of that particular "Rehab Brief and other 1 publications which 
may be of interest to the field 

In addition, the Evaluation Section Library provides techni- 
cal assistance to staff on specific areas in which there is a need, 
i.e., provides information on specific disabilities, resources 
which are available, etc. 

The use of the Evaluation Section Library is increasing 
constantly and has been very well received by the staff Th* 
following is a portion*of a letter recently received from a field 
counselor, "May I take this opportunity to tell you how very 
impressed I am by the service you are providing. One of the 
problems which has always faced the rehabilitation counselorm 
the field is the lack of current information regarding resources, 
prograrra^ program planning, etc., which could, if we hajjj jt;* 
^ake"uT7nT?Te~cffectivc in our service delivery. This is the first 
time, in my experience, that a variety of information is being 
made avaibble to use and I am finding ij enormously helpful 
and I sincerely hope that this service will be continued '* 



THE MICHIGAN MODEL EVALUATION UNIT - 
HALF-WAY THROUGH THE CONTRACT 

Robert *D. Struthers, Ph.D., Principal Investigator 

We are now halfway through our contract to produce 
"Comprehensive state VR program and po|,icy~systems through 
•model evaluation/ man ajenirent information support units." 
For us in Michigan it is nol a particularly comfortable status. > 
The novelty of getting the contract and starting the project has 
* passed, but most of the results we hope for have yet to be pro- 
duced. It secin's appropriate to list and discuks briefly some of 
thcV^ uses an ^ minuses The positives are listed first. 

The unit has been staffed (mostly), from an evaluation unit 



" . . work has begun . v on . .'. a measure of. , 

client changes which will fit the agency 's 

program evaluation approach . ' 
, % 

efficient and effective fashion than was previously the case. 
Three members of the PE staff are receiving training in using 
the terminal, and within a few weeks ( they Should be able to 
perform many operations on it. 

* > 

The unit has been 'able to utilize .a Wang Word Processor for , 
' a number, of applications. The word processor was obtained 
independently of the project but has proven extremely useful 
in preparing follow-up mailing*, tabulating and summarizing 
questionnaire findings, and in producing a variety of reports. 
It is particularly helpful in setting up tables so that data can be 
changed and tables*econstructed readily. 

The unit has gathered most of the data for the field testing 
of the Berkeley Planning Associates revised Federal Program 
Evaluation Standards It will soon forward the data to Bf*A and 
will write the state's report of its-experience in the field test. 

The unit has enlisted 22 rehabilitation facilities and 5 medical 
rehabilitation facilities to do the field testing of the* Facility's 
* Information System which was developed by Walker and Asso- 
( dates. The system is being considered for nationwide use by 
rehabilitation facilities. ThTe field test will be completed in early 
]981, and at that time Michigan will provide a report of its 
experiences and impressions. 

After many delays in the subcontracting process, work lite 
. begun by the University Of .Michigan's Rebihtation Research 
- Institute on several subjects of interest to the agency Among 
them are a measure of client change which jvilt/it the agency's 
program evaluation approach* and efforts^o utilize agency 
data to predict the e/fects oj^erving different client groups. • 

% WJiile it's reassuring to view the progress that has been- 
made, not all aspects of the project are positive. We know that 
we will meet the contract requirements, but we are not sure' 



ot one evaluatoi and one secretary, the unit has been expanded 
"to a unit leader, three program evaluators. twu secretaries, and 
a part-time graduate student. The original plans called for a 



that we "will solve some of the problems we have addressed nor 
that we will have m place the model evaluation system that we 
envision. It bothers us to be preoccupied with contract tasks 
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Edwin W. Martin. Assistant Secretary for Special 
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THE DURABILITY OF A MODEL ^EVALUATION UNIT 



WILJJAM W. JENKINS/ ED, D. 7 4 * \\ . 
PENNSYLVANIA" BUREAU OF - VOC^TiONAL ' REHABILITATION ' 

'* 7 ■ • . . ' ; 

The specifications for this contract^required the development of a 
' "Madel'V Program Evaluation and Management tnfonnation Support Unit. In this" 
context the term "model "-would "appear to have several levels of .meaning. «A 
model might be/considered *to be an. ideal of exqrc£>lary method of functioning. 
This was clearly one aspect, of the Rehabilitation Service A(3numstratipn ! s 
(RSA) intentions, since the six state* linits were to serve as examples that 
"given sufficient .resources ] St*ate Agencies* regardless of size, can develop 
■effective evaluation capacity* (U.S. Office of Humeri .Development Services,. 
1978) . However,, a model also has a theoretical ^ or scientific level of 
meaning which is to serve as M a representation of the i&d^tlying structure , 
of a prodess or* system. The system might be conceptual, ideal^ br -real *In 
general^ a model has a siirple anchor rranipulafable structure relative to the 
system i,'t represents. By nfeking explicit the implications of alternative . 
assumptions regarding key. relationships of , the issue of system under study, 
a model can provide a clearer understanding of these relationships" (U.S. 
General Accounting Office/ 1978) . We now recognize model construction and 
model description as an essential portion of the contract's work. The 
Bequest for Proposal (RFP) contributed to the process of model articulation 
by specifically ^requiring a continuing series of deliverables** on the concep- 
tual framework, operational tracking system, and evaluation plan. In doing 
so/ it provided both a > focus for many of the conceptual activities and 
parameters for them. ' 



Itiese Activities, at first glance, would appeal to corrprise a series of 
logical and rational functions which could be conducted, at a relatively 
abstract level/ .However, in an organization which already had an established 
Program Evalu ation Unit with, its own identity and history, such actions in- \ 
volved considerably more than we anticipated. The process of reconcep tual i z iritj 
the goals and nature of the existing unit toojc us into anextrenely sensitive 
and difficult-process . • In psychological, terms it required the development of 
.the self -conscious tp a. much greater degree than had previously existed. ^ In 
looking at the objectives, structure, organizational relationship, activities 
conducted, research strategies and the like, we began to encounter both 
personal and structural resistance that needed to be worked through. This 1 
process was .both demanding and time-consurtiing.^ Most of the first year of - 
contract operations was spent in conceptualizing, planning, and organizing 
the Msdel Unit and its initial work. .Since further deliverables are ' 
required in these areas throughout the 'contract, a significant portion of the 
-second year and, we suspect, third. year will also be consumed in; resolving 
pur "identity crisis". -Even after eighteen months of operation, the Resolu- 
tion is not yet clear 'to us. Unfortunately, the pressure of other activities ^ 
in the contract and continuing revisions in scheduling and work requirements - 
has reduced the tima available for reflection and. analysis of the model. I 
Despite these limitations, the opportunity to reconceptualize the "role of 
evaluators and information in vocational rehabilitation (VR) agency has led 
to significant ^alterations in our previous viewpoints a£ well as served to 
stimulate ,2taf f profe|pional development. 
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• - ' ln< developing a K>del Unit, one £ S^tSt Possible . 
structure this unit in such 'a manner '^^t^uW havetne grea ^ ^ 

-iinpact-on-the.agency's^FOlicy making, P^ n ff ^ liakages"„ it _had_ 

^required that the un j^^^f ru^ctSSlithS'the agency, 
with the sections responsible' for each of _tnese -organization 
S attesting to situate ^del ™*^^^J°%iter the 
ana to develop a "model" structure, it .^J^ o ^f^ aanizationa i ■ structure 
characteristics of the ^ a ^J^^^^' < SS^ract in the 
and hierarchy. T*e PennsyiVan^ agency T^^W^s^llion 'dollars) and, 
category of a large agency^ecUon ^^Pg Saracteristics of a' large 
with its to^-WS^S^SSSSl theory and structure .(Porter, 
bureaucracy. Tte literacnre un cu-yo^x"* a 4-+.pn«ts to maintain its . 

SFS^ suggests that a ^-^^f^tSt^S Pennsylvania 
exists state' of eguilibriun. *"*™9^ ^ ™J ^ nec essarily - 

• agency is reaarkaply Oxg ^dynamc for ^ z ^ h could have 
possess sorre degree of ^.^f ^^'rev^Stfonary jrethods/ the exist- . 
bSn initiated in the ore^ilinSana-geneht philosophy 
iiig-ErogramB^^^Unxtas v^las .^^^f^ct. jffiis logically 

after the contract perxbd. L.^ , 

In raving tt. *i tot STa'SS^ 

for th* unit to ""^f^' £°? l5?<£SS» to the 

torn the a-jency ait xt «=^*f ^ taw Wrea recently, 

aticnTcn^catienal--Eehabilitation i?/-> • -^ v _^ ith - the federal 

Jhi« ftmction within nost agencies. However, E^f^^"™ A ^ ' 



^£*toS£d» nost aggies However g^^nT^" 
PrSram Standards, the ambiguity of ^^^^S to continuing ■ ' 
defensives of many ^f^^i^^S^m longer specifically 
resistance to its operation f/ a 1 ^^^t^f W^peraSohs and 
identifies program -evaluation as a requirea -, nsterit y of low state • 

^ its Sace ^igu^ • , 

budgets at, this -tune and the forecast ^ even grea agencies, 
^nfutility and cost-ef f^ness o ^^f^^SiSty beco^s a^- • 
is being closely scrutinized, ^tionor even ^ re levant for a Model 
issue for all Program ^f^° er ^^^ortunities for the model 
Unit. However, this crisis al^pof^ JfiSSS*. aTability to improve 
to clearly prove their utility ^ **Ti^Sh pivotal times, 
operations and make them more efficient in sucn pivux^u. 

At «. inception or-fhe «°~ ■« S^^KffiEZ^ .X 
section «hich had e^lved several V^^SjfSd'SrSiJB professional 
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served as Case Evaluators for field studies. Both the Evaluation and Manage- 
_ment Information Suppo rt functions of t h e agency were conducted within the 
I sectionralthough data processing was handled by a^difrerentTunit. The ■ ' 
secfeon was supported by an 'IBM 370/158 computer through on-line terminals 
in all field offices and central office, with the section given high priorit/ 
for use of these facilities. However, there was not any capability within '" 
this system for high level -statistical analysis through a package computer- 
ized system. In the organization hierarchy, Program Evaluation Section was 
removed two levels from the agency's director. 

• From the' beginning it wa% felt that the model unit's impact and "chances 
of its work being durable would be substantially increased if it could' be* 
closely tied to the existing Program Evaluation Section. By doiSta this, the - 

- existing section's reputation, linkages, technical facilities and!dther " 
resources could be utilized from the onset without the model unit having to 
do all .the original developmental work. Therefore, the Model Evaluation Unit 
was made a separate unit within the existing Program Evaluation Section. The 
dual structure has the advantages of providing centralized information and 
budget control while giving special emphasis to the contract activities. 
The dual structure also provides research and field support at' f an indepth 
level while maintaining flexibility and the experimenta* quality of the unit 
without formally conmitting it- to a particular organizational structure. 
Uider this system the Principal Investigator is responsible for conceptuali- 
zation, .research and technical direction as well as coordinating staff efforts 
on activities. .The Project Manager is also the administrator of the Evalu- , 

•ation Unit and provides administrative and. budget direction. The manager also 
ls responsible for coordination between the unit and regular section and the 
agency. .? •- 

t - 

The REP requiqed'that^the unit be situated organizationally "so that it ^ 
is directly responsible and reports directly to the agency director, who is 
also the project director. This innovative and essential feature has been 



particularly importan t in facilita ti ng direct and" frequent coimi unication 
between the Model unit and the agency's higher organizational levels, and 
has provided additional authority to the unit's' staff during the first year. ' 
This has' also provided valuable insight into the policy and management < 
decision-making process that would -not have been attained otherwise. 
Additionally, a coordinating coimuttee composed of central office staff and 
administrators was organized- to provide input and to assist in planning and ' 
coordinating. efforts of the unit .across, departmental lines. Overall, the 
placement of the Model Unit within the agency -and the development of its- 
linkages with other. sections has been designed to both increase- its effective- 
ness and to structurally integrate it into the existing organization m the 
most corriucive»manner. Throughout the contract, resources, consultation and 
training opportunities, have been shared between, the 'agency's' regular and 
model evaluation -units with the deliberate intention of gradually merging 
their skills, activities and functions so that units may have a gwisolidated 
single- identity by 'the contract's end. - * w • 
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several other factors were designed .to insure the unit's tapact and 

duxabSS « • «» - to HT> VSSSSCSfw 1 

ov^niri iimediatelv begin to produce work .that is of junportanse to cne^agw^x 

f ^ SLriSced toS S evaluation techniques and agency operations 

have an. iitpact. • Such factors were given ^ r ^S as tortant, 
staff for a nodel operation and are now clearly recognized as ompo^ 

"our experiences to date in constructing * "^f"^g^ Snsuming 
. suggest thaVthis process is considerably nore dlf ^* andtu^ns™ 
KThad been previously thought. -In particular, the need to focus on me 
derignairf structural characteristics of the unit, ^th to ^cgase its 
dSaSlity and to experiinentally find- the best J^^^J^SSt 
SSSres more conceptualization, organizataonal support, and gr^cipanr ^ 
SLentand flexibility than was anticipated. ^WS^giS^ly. 
participants and the organization, however, ^^^P^^^^ntinuing 
^ r Ser P than- imagined. The ^-tem ^eof S^SgSSl^ESSrf a 

impact- appears to fee a necessary " prig ) Hiai -' 

Sdel" evaluation unit if it,, in fact, is to endure. 
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Uation prior to attempts at implementation of any evaluation- 
program o^ project - Mircly no one who was listening will ever 
erfhere! Mr. Markisolm was followed by a presentation on 
field test experiences by members of the Delaware DVR staff. 
"Aside from experience itself there is probably no better teacher 
than someone who has "been there** and Martha Jackson, Case 
Review Specialist. Larl C. Tuberson. Administrator. Barbara 
Bennett, Casework Supervisor, and Robert Snider, Facilities 
Specialist, ha^ve all "been there."* They shared with the groUp 
their experiences with the San Diego Case Review Schedule, 
the New t:\aluation Standards, and the Facilities Information 
System. Those who listened closely will be very glad someday * 
Jhcydttl! 

The final session of the conference on the afternoon of the 
second day was one in which the representatives from each 
of the participating sister states briefly described evaluation 
activities in his or her own VR agency. The hope is that there 
wiB be more sister state conferences in which eachjofthe sister 
states can share more fully with the otljcxs its successes and 
failures in program evaluation so that successes can be dupli- 
cated and pitfalls avoided -which means everyone wins. Future 
newsletters should tell the story. (They may also tell the story 
**of some other sister states. Harry \V. Guise, Richard Cohen and 
William W. Jenkins. lid.D.. of the Pennsylvania [another MEU 
stale but a much larger siate[ Bureau of Vocational Rehabili- 
tation, impressed by the concept of the sister states conference, 
attended the conference to observe and consider the applica- 
bility of the concept to larger states.) . + . 

For now it's ''Thanks. Guy Veach for your brainchild." ^Vell 
done. Mr. Veach and Delaware DVR." and a special titan k^to 
clerk«t>pist Jcan'Gibbs of Delaware DVR for your behind-tnc^ 
scenes activities, your work on the registration desk, the photo- 
copying of material. :.nd all the numerous other chores you 
(lid for everyone that helped make the conference the success 
it was. 



GRANTS - CONTRACTS 
BUDGETAKILY^FEAKING 1 

Harold ^eit7et> 
Pennsylvania Bureau of 
Vocational Rehabilitation . 

Lucfcv vou! Yon own Hour own home, one or two cars, 
maybe a boat, a summer house at the lake - M~ are you just 
"buying" all those \^ondcfful things? If so v yoi arc, whether 
you know it or not. a money manager, a Budget and Finance 
Specialist, because, before buying those things, you had to 
decide to purchase them, and (II yon arc like the rest of us!) 
you had to decide on the basis of j limited supply of money 
available to you. So you had to weigh your needs and wants # 
and choose which were most important to you. You had to 
decide what you were willing to do without or with less of 
in order to have something elve more important to you. Alio* 
you have to keep tuvk»of your money to nuke sure you atcu't 
Spending too much or wasting money on things that arc of less 
Importance to you. 



Government contracts or grants, usually representing large 
and impressive sums of money, often generate the psychological J 
effect of euphoria or that one's ship has come in. But, in fact, 
has it really? 

From the outset, the initial decision to bid on. or apply for 
participation in a government grant, (he administrator must 
begin thinking, planning and prudently weighing the benefits 
to be gained. 

Mindful that each expenditure must be well placed, there 
arc numerous considerations and preparations that must be ' 
undertaken. 

Time consumed by staff members and accurate reporting 
of it arc essential. Time and money are one and the same, since 



"The forms, and documentary reports ; • • 
now contain statistical data and information 
that clearly illustrates^not only how budget 
money has been used but also where its use 
has been most effective. 3 ' 



they both represent dollars. Travel and travel related expendi* 
tures such as lodging, pe^diem, etc. must also be made a matter 
of record and budgeted wisely. / 

In order to maintain a clear and easy to understand system of 
budget maintenance many contracts require a style of reporting 
that is unique to the needs of a particular project. # 

The Commonwealth of Pennsylvania, presently a participant 
in the MEU contract, has designed a series of reporting docu^ 
ments and forms that arc used for the express purpose of 
trackings various activities and accompanying costs from the 
beginning to any given tune throughout the life of the grant or 
contract. From these documents the administrator can gauge 
the financial posture that has been maintained and determine 
if the budget is adequate to continue planned activities or 
whether additions or reductions should be considered in various 
tasV categories. The mere fact that funds have been made availa- 
ble giould not encourage the "use it or lose it" style of planning. 
For pcakcffcctiveiicss in contract results as well as effectiveness 
from the standpoint of spending, total effort should be applied 
to all stages and all tasks. 

The forms and documentary reports designed for the Com- 
monwealth of Pennsylvania now contain statistical data and 
information that clearly illustrates notonjy how budget money 
has been used but also where its use has been most effective. 
This, without any doubt, is an aid in directing and adjusting 
future activities related to the grant or contract. - 

At reporting time there is no easy way to acquire and 
assemble alt the pertinent material, papers and information 
necessary for a clean and clear picture. These arc gathered only 
through on-going, persistent and tireless effort. This is especially 
true of projects of unusual sue or scope. 

The Pennsylvania project reporting budget is designed to 
develop separate stages of its system and to "feather** each step 
into a succeeding step that finally enables all steps to be joined 
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into 3 coiicbc report that is easy to read and to comprehend. 
Because uf this excellent system Penns) Kama's contract budget 
Enjoying good health and with indent watchfulness shuuld 

ain healthy. ^ 
• As stated earlier principles applied tn running one's .own 
household are nut unlike planning budgetary expenditures 
invulvcd in government contracts and grants. It is merely 
necessary to use tittnust cautiun and care nut to engage tn 
^excess" with funds furnished for a contract or grant anymore- 
that you might buy two or more bicycles furVeach of your 
children just because there appeared tu be "lots of money to 
get rid ofT If you arc interested in more information on how 
Pennsylvania docs tlus write tu Harold Seitz'or at the Pennsyl- 
vania Bureau of Vocational Rehabilitation, Labor and Industry 
Building. 7lh and Furstcr StrcetsTllarrisburg. PA 17120. 

OREGON VOCATIONAL REHABILITATION 
DIVISION REORGANIZED. - 

— - * 

The Oregon Vocational Rehabilitation Division has recently 
undergune reorganization. Tins changed the "conceptual frame- 
work" required by the MEU contract. At the same tune, tins 
MEU contract requirement affected the process of reorganizing, 

Oregon's MEL 1 Project Principal Investigator, Neil Sherwood, 
was a member of a reorganization study team created by the; 
Division's Administrator. MEU consultant, Dr. Bruce Gates.4 
served as a consultant to the study team and the Administrator. \ 

Neil Sherwood expects to write an article describing the* | 
ents. processes and experiences involved in conceptualizing*-^ 
und urganizing the Orcgnn agency. This article will most likely 
appear in the Manuscript of Special Observcd,Activity (MSOA) 
to be published at a future date. v * 

A desired outcome uf the reorganization is to broaden t(ic 
local impact of the MEU Project, this may result frum twu 
' changes. Hie first change was- to organizationally integrate 
management information support fonctions with automated 
data processing fonctions v The second change was to integrate 
program planning and evaluation fonctions within another 
organizational unit. For more aboot this._watch for fotore 
installments. 

MICHIGAN'S MODEL EVALUATION PROJECT 

Robert D. Struthcrs. Ph.D. 
Principal Investigator 
Michigan Bttrcatt of Rehabilitation 

Michigan's venture into cuntracjl.md continues wfth mixed 
rcsolts. We're beginning to sec the cunclusiun of some nf the 
contract tasks, bot others arc still proceeding slowly. We arc 
anxtoos to torn nor attention to some new issoes which m.c 
think will make a eontnbution to the field. Here arc sonic 
random comments on different aspects uf the contract. 1 

Part ta/xt tiny in the t**tnevt. The project has been an educa- 
ion in duim: things the federal way. We have learned what it 
means to prepare pro|H»j)s. tu establish budgets, tu write 
cunccptujt frameworks, and to write |»lan> fur the various 



parts of the contract. Unfurtunatcly, becaose of' the many 
bureaucratic snags much of uu^ efforts have seemed like busy 
work. We have spent many yoors on reports that arc now uf 
ILklc use because of changed conditiuns, and we have spent 
Injurs in disucssion of issocs which were finally resolved with- 
out regard to the deliberations. The project is moving bot 
we're m a period wh'cn sumc of our initial expectations arc 
being revised and products are not emerging as rapidly as we 
wish* they were. > * 

rVciV Capacity for Data Processing. In previous newsletters 
we repdrted on new PE personnel that the Michigan agency has 
been able to add as a result of the project. Another significant 
gain in capacity has b«en ubtaining an Anderson-Jacobson 
AJ 832-30 computer terminal located within the PE unit and 
linked to the computer network which serves the three major 
universities in the §tate - Michigan, Michigan State and Wayne 
State. The terminal has been leased fur S177 per month. Part 
of the rent can be applied to a purchase if desired. The price of 
the terminal is approximately S3.000. in addition to the cost 
of the terminal, we expect to use about S2 ,000 in computer time 
this year and we also purchased training to use the terminal. 
We have trained three people in utilizing the terminal and 
becoming familiar with the variety of statistical packages 
which are available. Training-was provided by the University of 
Michigan's Rehabilitation Research Institute. 



" . . . the figure for the nutnbef^bf' perso ns 
rehabilitated per dollar , spent is probably 
the most useful single indicator of effec- 
tiveness from the perspective of agency 
administrators" 



• Use of a Word Processor. In. the fall uf 1979 Michigan BR 
y^Jbtaincd a Wang Word Prqjrfsshjr, System 5. Model HI. The 
word processor was obtained independently of the modeievalu- 
atiun project, but using ir has irfoven to be a great help to the 
r unit. The four major applications fur a wurd processor are 
• revisiun typing, repetitive typing, data processing, and statis* 
tical typing. We have used the J^ang Wurd Prucesso/ to prepare 
research articles, questionnaires, eude books, grant proposals, 
and a computer manual. Wc^arc huping to use it to reptacc the 
"subject canT in cataloguing our PE literature. 

We have fuund the word piucessor particularly helpful in 
setting up and revising numeric tahles, and the Wang has suf- 
ficient capacity fur arithmetic computation that we were able 
tu use it fur simple data pruccssing before we ubtaiocd ouf • 
data processing terminal* L . 

The Federal Program I. valuation Standouts. Our part hr the 
field testing of the Federal Program' Evalu.iuun Standards is 
drawing to a close and we will be submitting our comments 
on the, stanUards and the procedures used in the field testing 
them in the near future. It appears tu us that, in general, 
the standaids arc feasible fur use as currently planned by 
USA- However, additiuital wurk by. the developer. Berkeley 
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CONCLUSIONS 

• How does all AIS tcUvtty relate to the primary mission of 
the MEuVojectf-Aflmittedly, from one point of view BKA 
and RSA have been the primary beneficiaries of the revised 
standards field-tcst to date. HowevV the field-test has returned 
«me Important dividends for the MEUs as well. First, .1 has 
provided an excellent opportunity for the MEU staffers and 
state, agency managers to think 'through the whole notion of 
standards both conceptually and methodologically and m urms 
of how they apply to managerial decision making processes 
within their State agencies/Secondly, the field-test has proved 
an excellent traimr)* laboratory, not only for new MEU statt 



members (some of whom have come from other disciplU 
and have had little contact with the world of vocational reh 
bilitation), but also for other staff members from aU levels 
and divisions of the state agencies who have had little contac , 
with program evaluators and program evaluation. Finally, Out 
certainly not least importantly, working through the data 
elements worrying about their accuracy, validity and reliability 
and arguing about their implications for agency decision-making 
has reminded us forccfulljfMhat the ultimate objective of the 
MEU project is the improvement of services for clients. If the 
standards field-test contributes to that end (and I for one 
certainly believe that it has and will), then its benefits will , 
have far exceeded its costs. 



IV. MEtJs ~ formation and operation 



\ 



MEU STAFF PARTICIPATES 
IN STATEWIDE STAFF MEETING % 

One VR agency that recognizes the importance and value^ 
of a good evaluation program and is taking full advantage of 
its Model Evaluation Unit is the Mississippi Vocational Reha- 
bilitation for the Blind (MVRB). In addition to haying been 
isked to assist in planning MVRfc's Statewide Staff Meeting. . 
the MEU staff had a very visible role m tfie actual conduct 
of the meeting which was held September 15, 16, and 17 ,1980 
in Jackson, Mississippi. The meeting brought together all 
Regional Directors, Counselors, Liaison Counselors, and Coun- 
selor 'Assistants employed by Mississippi Vocational Reha- 

bilitation for the Blind as well as fcy administrative staff. 

Dr 'Martha Walker, Professor and Coordinator of Rent State 
^University's Rehabilitation Program, served as moderator for 

the three-day session. 

At the specific request of the Director of Mississippi Voca- 
tional Rehabilitation for the Bhnd, the analysis and findings 
resulting from the Mississippi MEU's participation in testing 
the San Diego Case Review Schedule were presented by the 
MEU Principal Investigator. The agency's strengths and weak- 
nesses as revealed by the pretest of the Standards were related 
to a thorough review of the rehabilitation process covering 
Statuses 00 to 10. As issues were raised in an attempt to up- 

" grade the skills of conference participants, .relevant findings 

' oftheCRS were noted. 

During its initial year of operation! the MEU participated 
with Mississippi Vocational Rehabilitation for the Blind's Pro. 
tram Support Unit in conducting the agency's annual Case 
Review/Quality Control Study and in preparing the subsequent 
report This experience enabled MtiU staff to contribute to 
the training session information garnered directly from agency 
05CS in such areas as case documentation, determination of 
"client eligibility, and federal and state mandates regulating 
what may occur in Statuses 00 to 10. 



Mississippi Vocational Rehabilitation for Jhe Blind has plans 
for two additional three-day training sessions to address Statuses 
10 through r 12 % It is very probable that the MEU wiUbe asked 
to assist in the planning and conduct of these sessions. , 



EXPERIENCES A^P IMPRESSIONS OF A 
CASE SERVICE EVALU ATOR , 

by Paul E.^Saupp 
> 

As a fust-line Supervisor with the Pennsylvania Bureau of „ 
Vocattonal Rehabilitation (BVR) in a unit consisting of two; 
specialty, caseloads, one providing services to psychiatric clients 
(behavioral disorders) and the other servicing SSD1-SSI bene- 
ficiaries, 1 welcomed the-chancc to apply for the position of 
Case Service Evaluator (CSE) when the opportunity presented . 
itself three years ago. The job would be a challenge I thought . 
- it was a new field both for me and for the BVR which had 
only initiated the case review process a year earlier - and it 
would permit me to acquire an in-depth perspective of the 
entire rehabilitation process. I was not disjointed As a 
Case Service Evaluate! 1 have been challenged and I have 
learned much. # ^ 

For administrative purposes the Pennsylvania BVR divides 
the state into four regions. In each of the four regions a CSE 
is responsible for evaluation activities. My particular bailiwick. 
I is Region 111 to , which I was assigned after ^tensive training 
by one of the other three CSEs. Geographically this is a rather 
large area of farms and small (o medium size towns with a total 
population of Some 1,654,^73. , ' 

The case evaluation proccHm Pennsylvania includes monthly 
meetings of the four rcgionakSEs and the Administrator of 
the Evaluation Section oj the BVR. These meetings are used to 
develop case revww sheets, question .uterprctation.avW provide 
guidelines for each review. Lively discussion occurs during these 
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sessions as each review item is examined closely and boiled 
down untU consensus is reached. I find this item analysis very 
appealing since the give-and-take among the participants results 
In a feeling of satisfaction with the accuracy of the finished 
product - a feeling that the close scrutiny of each item has 
resulted in a synthesis based on the best thinking of the entire 
evaluation- team. 

After a consensus on the case review package is developed, 
the results are "put to the test" to determine inter-cvaluator 
reliability by having the CSEs review a common casefile. If^ 
the results indicate acceptable reliability, i.e., if the e valuators 
exhibit a high degree of uniformity in respect to interpretation 
of the review items, the p ackage is consid ered a finished prod- 
uct To me, this aspect is thCheart of the evaluation process . 
since without a high degree fef inter-evaluator reliability, sub- 
sequent stages of the process will be meaningless. . , 

The next stage is field testing the finished product. This 
consists of selecting an identical number of cases to be reviewed 
by each evaluator i^nis particular region. The*results are.evalu- 
ated at another meeting of the Case Service Evaluators and 
Administrator tne Evaluation Section with particular empha- 
sis on any unforeseen problem areas that were uncovered in 
the review. Utese areas are refined until a consensus is again 
reached by the evaluation team. This stepMs also^ry important 
\ to the case evaluation process since, again, it assures high inter- 
cvaluator reliability and uniformity in interpretation of the 
case review items. 



"Without the necessary follow-up, the time, 
effort and money expended on a case review 
couid end up being time, effort and money 
wasted* 



After agreement is reached at the two stages of; the evalu- 
ation process, the reviews^ conducted in the district offices 
by the Case Service Evaluators. If the approach of thi evaluator 
in case review 4 is not punitive, if it is positive in tha: the evalu- 
ator is sincerely interested in attempting to help the district 
administrator obtain a^nore efficient operation,, the district 
administrator will likely perceive the evaluator as an asset in his 
planning. Likewise, the role of the evaluator will be perceived 
as threatening to the counselors if they arc* susjMciotfithat 
they are being judged on an individual basis. It is r.cccssdry to 
convince counselors that the case reviewer is interested only in 
patterns of weakness and strength in the offilas a unit. In 
addition, it is mandatory to make^crtain ^Supervisors do 
not use individual counseipr results ffom the review m a punitive 
way fsucli as in the annual rating of the counselor). This would 
defeat the real purpos^of the evaluation. 



After a district office evaluation has been completed, a Case 
Review anti Evaluation Conference is arranged with the district 
administrator, regional administrator or assistant, and distri* 
office supervisors when feasible.The purpose of the conference 
is to examine the results of the case review including strong 
and weaK areas, identification and solution of problem case 
service patterns, and clarification and modification of existing 
case service patterns where needed. This conference is of the 
utmost importance because it is here in the interaction between 
the participants that the district administrator who is interested 
in improving the operation of the district office/can get new 
f ideas. As patterns of weakness are identified, the likely causes 
for these weaknesses are explored and possible solutions sug- 
gested Patterns of strength that can be built upon are also 
examined and discussed. If, however, the district administrator 
another participants do not have confidence either in the 
evaluator or the findings i>£ the evaluation, or if they are adverse 
to change in general, the meeting may well be of little value, 
thus, it is most important that from the very first the evaluator 
work to establish rapport with the district administrator and 
others, conduct himself or herself in a professional manner 
always, and be well informed on the subject of each review 
so that the district administrator can develop confidence and 
respect for the CSE. The gfeater the involvement of the par- 
ticipants on the whole process. the greater is the probability 
that the' conference will be successful in generating necessary 
changes within the district office. 

In addition to the Case Review and Evaluation Conferent 
another very important aspect of the case evaluation process 
is the follow-up. By follow-up I .mean a systematic" method of 
determining if the weaknesses detected in the case reviews and/ 
or discussed in the Case Review arfd Evaluation Conferences 
have been corrected or are being corrected in the district ofTiccs. 
My greatest concern is that not enough emphasis is placed 
. on this very important aspect of the case evalua^on process. 
Without the necessary follow-up, the time, effort arid money 
expended On a case review could end up being time, effort 
and money wasted. t 

My three years as Case Service Evaluator have been very 
rewarding. I have gained a greater understanding of the rehabili- 
tation process, Rehabilitation Act of 1973 and Amendments, 
the Federal Regulations and Procedures Manual, and policies 
and procedures in general. Convinced of the value of and (he 
need for evaluation in Vk before 1 became an evaluator. my 
three years as a CSE has reinforced this conviction . Evaluation 
is vital to the good health of VR and I am happy to report that 
the Commonwealth of Pennsylvania has a coherent and effec- 
tive evaluation program, (although some areas need strengthen- 
ing) that is an excellent vehicle for evaluating and monitoring 
the service received by clients as well as compliance with 
Federahind State regulations. 
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PROGRAM EVALUATION MEETING IN 
REGION V 
Robert D, Struthers, Principal Investigator, 
Michigan MEU 



As part of its dissemination activites the Michigan 
'MEU co-sponsored. a meeting of program evaluators 
from Region V in September 1980. The conference was 
cc-eponaored with the Region V- Regional Office and 
the University of Michigan's Rehabilitation Research 
Institute and was held in Ann Arbor. The meeting was 
particulary welcome in Region V as there had been no 
opportunity for program evaluators tp get together for 
several years. * 

The first day of the conference was devoted^riman- 
ly to a description of Michigan's model project ac 
tivites, and the second day was devoted to presenta- 
tions from the other states. Michigan's presentations 
consisted of an overview of MEU activities by Bob. 
Stmthers, followed by a description of the field-testing * 
of the proposed revised Federal Program Evaluation 
Standards by Bob Richardson and of the proposed 
Facilities Information System by Geri Hansen. Jim 
Nnttall then "deTscribed the unit's experiences in 
bolstering its analytic capacity with a computer ter- 
minal linked to the Univeristy of Michigan's computer 
center. 

It was evident from the presentations on the second 
day that progress in program evaluation is by no 
means limited to the model units. All of the states 
reported on projects that were of interest. Minnesota 
describedtts statewide survey of the disabled which is 
baaed on functional limitations, plus case review in* 
diemtors which they use. Wisconsin is in the process of 
developing a totally new information reporting 
system* Indiana„baa recently completed a study 'of its 
on-job training activities. Ohio has a process for ad- 
dressing and sending out follow-up surveys by com- 
puter, and Illinois has a number of approaches to the 
evaluation of special projects which were of interest to 
the group. > 

la addition to the state reports Terry Coribur. and 
George McCrowey of Region V described the Abt pro- 
ject en the revision of the national management infor- 
mation system. Don Harrison and Ralph Crystal 
reviewed the recent activites and publications of the 
Ualvert ity of Michigan's Rehabilitation Research In- 
stil**, and Rick Nida of the West Virginia Rehabilita- 

•t # *Ji r * etrth V d Trainin K Center explained how 
lUfWa 111 hss maintained, continued communication 
«M iaurtetioo among program evaluators through 
ikmif TE Forums." 



Duane Sermon of Minnesota accepted the position of 
chairperson of the Region V group and it is hoped that 
la the future semi-annual meetings can be held. 



302 'a 



SECRETARY REFLECTS -ON MEU IN PA 
Zelda M. Peters 
Pennsylvania Bureau of Vocational Rehabilitation 




Yippee! We were awarded a contract! So what? New 
words and phrases were being tossed around, and it . 
sounded like a big paper chase to me. How could we 
produce a model evaluation unit? Just what is a model? 
Well, we soon found out; an<J it Wned out to be 
•enlightening to all of us. We found out that our unit 
was *i integral link between our Bureau and a whole 
host of "users** $ut there who needed the information 
we could provide. 

As NASA has a special language or jargon, peculiar 
to the Space program, the MBUs have their own 
special jargon which* is becoming more a part of the 
everyday language of qur unit. 

« ' y 

So, "dissemination and utilization of information H , 
"management information system", . "contractors", 
"products", "deliverables", all took on new meaning to 
most of us. Not to mention the increased workloads. 
Those typewriters really took a beating. And the 
meetings with R&T personnel, listening, absorbing 
what we could, and at times contributing some'small 
bit of information or some idea that we had, gave us a 
vision of what we were supposed to be working for. 

"Deliverables" now that's a good word. We learned 
pretty soon what they were, and the purposes they 
served. We saw the outline cfour 3 yeanprogram tak- 
ing place. It seemed that there was no end of jargon 
and 60-cent words*"flying around our Section, and we 
almost thought they were newly invented. The noise of 
the typewriters, calculators and voices discussing 
various aspects of our goals was deafening at times. 
We felt at first like small ships- in uncharted seas. $ut 
when all the fog had cleared away we saw something 
very grand emerging from all this that we had con- 
sidered "gobbledy gook". This model evaluation unit 
stuff was really okay. * 

• We noticed, and still notice, a difference in the com- 
munication between our unit and other units in our 
Bureau. We began to get an idea about what the other 
units' goals were*in relation to our own. We started to 
feel like part of a giant organization with united pur- 
pose which was starting to roll with such for.ee that 
nothing could now stop it. 
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To one whose "niche" in an organization consists of 
typing letters, memorandas, reports, charts, and the 
like there is little time for, or interest in, the activities 
of other units whose work activities are unknown and 
seem far removed f^om thegoals of the unit in which 
that person provides "technical assistance". You see, 
there I am beginning to take to that jargon we found so 
new and strange a while back.^ 

To be specfl? about a few of the things that I see oc- 
curring from where I sit at my hot little typewriter - 
we are cognizant of the part other units of our Bureau 
play in the attainment of the overall goals of our 
Bureau. There is a new awareness that we all have 
specific roles to play. So that the completed project is 
properly balanced and provides the products that our 
users require in a timely and orderly manner we must 
complete our own projected goals as our timetable 
prescribes. 

"It is fascinating to be part*of this model evaluation 
unit, and to see the good that it has done for us in our 
scheduling of activities and goals. We're coming down 
the "home stretch" and it has been a rewarding ex- 
perience. 



DATA DICTIONARY 

The Oregon MEU is continuing its efforts to enhance 
the agency's data processing capability. Oregon Voca- 
tional Rehabilitation Division (OVRD) recently con- 
tracted with the Oregon Executive Department's Data 
Systems Division to begin a systems analysis to define 
the requirements of a comprehensive accounting and < 
purchasing system. This system wiJJ be integrated 

. with the client data system developed earlier. 

< "» • 

The development of all these computer systems has 
increased the complexity of data management. To ad- 
dress the resulting problems the MEU is developing 
specifications for standard naming conventions and 
formats to create an agency data dictionary using the 
software package Data Manager. 

Data Manager, developed by MSP, United Kingdom, 
is a user 'oriented product for* creating, maintaining, 
and questioning a data dictionary. It does not contain 
the real data itself, but contains data about data and 
permits documentation of the characteristics of each 
data element as well as documentation of the processes' 
that act on it For example, after the dictionary is fujjy 
developed and all OVRD data elements, files, pro- 
grams, and systems are documented/ it will be possible 
to query the dictionary and determine every use of a 
particular data element* 



FMIS UPDATE 

The Facility Management Information System 
* Evaluation Panel met in Chicago, Friday, April 24, 1981. 
The NARF project staff together with Walker and 
Associates' representative presented preliminary fin* 
dings of the FMIS field test experience. 

The response to the evaluation questionnaire has 
been very good - a 70% return. The NARF staff, in 
its efforts to strengthen the validity of the evluation, 
made follow-up phone calls and individual contacts 
with many of the field test participants. 

The NARF/FMIS staff feels that a good deal of cau- 
■ tion must be exerted at this time in dewing conclu- 
sions about the FMIScThe reasons for this caution, as* 
revealed by the examination of the evaluation, are: 

( • All sources of data do not necessarily provide the 
same general results. For example, the question- 
naire responses tend to be more positive than 
, negative while the commend in the state > 
- meetings tend to be more negative than positive. 

• Initial field test data displays were prepared 
from incomplete data to demonstrate some of the 
typet of displays that could be prepared. Due to 
incomplete data and lumping together of .dif- 
ferent types of facilities and programs the field 
test data displays were skewed and misleading if 
viewed for the data itself rather than as examples 
of types of displays. This appears to have con- 
tributed to some of the negative evaluation feed- 
back. 

• It is evident from the experience of the field test 
that the variety of facilities, programs, state 
agency approaches, outcomes, and even 

' disabilities is greater than was perceived at the 
beginning of the development of the FMIS. It ap- 
pears that FMIS may bo appropriate to some of 
these and not others. Determining which and to 
what extent will require further analysis. 

Given these findings it was evident by the April 24 » 
Evaluation Panel Meeting that more detailed analysis 
of the data was neefle<(. The Panel concurred in the 
recommendation of staff that this be done. The key ae- 
v tivities in this analysis, which are now taking place, are 
the following: 

• Additional sorts of facility data by characteristics 
sueh as facility size, type and utilization., 

• Slore detailed item by item .examination of ques- 
tionnaire data. 




WORKING RELATIONSHIPS AS A FACTOR' 
IN THE DURABILITY OF PROGRAM EVALUATION 
*. , •' Donald E. Hoiiler 

P«nn*ylvantalBureau of Vocational Rehabilitation ' 
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One of the keys to the durability of an organizational 
unit is its ability to relate to'* other units of the 
organization^ a successful manner. The Pennsylvania 
model evaluation unit (PA MEU)has expanded its rela- 
tionships with other central, regional- and district of- 
fice units throughout the State since" the inception of 
the contract A major role of-the Program Evaluation 
Section prior to the awarding of the contract was sup- 
port activities Xo the entire Bureau. Contract work has 
heightened this involve/nent. Examples of such on- 
going relationships are:*" 

1. Case Review Process - This requires input from 
literally all segments 6f each organizational unit of the 
Bureau. This input assists in the determination of sub- 
jects for case reviews, e.g- homemaker closures, 
suitability of choice of vocational objective, alcoholic 
case reyiews, etc Additionally, top management's 
utilization of review can result in policy modification 
and can create an inter-active environment. 
♦ 

' 2, Federal Standards - Compilation and analysis of 
statewide data for this purpose "requires cooperation 
and consultation from the district offices and manage- 
ment at all levels. • 

8. Facilities Information System (FIS) - This work ac- 
tivity involves relationships with the Facilities & 
Grants Management 'Section in order to develop and 
implement a system^ * « 

4. Technical Assistance Center (T.A.C«) - The collec- 
tion of journals, periodicals, books and otherliterature 
has enhanced'our capabijity to provide sound informa- 
tion to Bureau'staff. In just over one year, almost 200 
requests for information were filled. Bequests 'are 
received from all 15 district offices and all sections of 
the central office. The range of requests is for informal 
tion on everything from disabilities and program 
evaluation to management, counseling, and job place- 
ment . ^ 

A specific example of Pennsylvania's program 
evaluation supportive role in our agency is recent 
work on an evaluation plan for .the newly emerging 
' Client Assistance Project (CAH 



The Program Evaluation Section (PES) first heard of 
t!w existence 3T CAPs through a "Rehab Brief 
disseminated by the University of Florida's Rehabilita- 
tion Research Institute on November 16, 1979. As a 
result, the PES contacted the Rehabilitation Serviced 
Administration in Washington. DC. to access a listing 
of operating CAPs. Phone contact was initiated with 
15*20 CAPs asking for their annual reports and forms 
used. The responses flowed into the PES and were log- 
ged into the Technical Assistance Center for future 
use. 

In April of 1980 the' Grants and Facilities Manage- 
ment Section (GAFMS) began gathering information to 
request Federal funds for establishment of a PA CAP. 
with a pilot study in 2 district offices and one office .for 
the, visually handicapped.. Information was 
disseminated from the T.A.C. to GAFMS in support of 
the proposal* 
%. 

yhe PES assisted in the development of the evalua- 
tion plan with presentations to district, regional! and 
central offices, and, to the CAP staff. The purposes of 
the 49 meeting were; (1) to^develop an awareness and 
understanding of the role of participants in the project; 
(2) to increase understanding of the evaluation plan 
system and purpose; (3) to initiate a uniform procedure 
at the onset with an awareness that modifications or 
deletions cdujd be instituted when necessary; and (4) to 
elicit input for any preliminary revisions to the evalua- 
tion plan before it became operational on February 1, 
1981. 

Topical areas presented by the PES at the meetings 
revolved around a general overview of program 
evaluation, discussion of CAP program objectives and 
how they are measured, review of all forms to be utiliz- 
ed and* their flow. Various audio-visual techniques 
were used to*promote discussion and comment. 

Further PES activity in* the CAP will involve 
i monitoring of progress, collecting data, measuring out- 
comes, analyzing data, and .disseminaton. Frequent 
PES meetings, are pjanned with Bureau field, central 
office and QAP^staff. 

* A high priority of the PES, in support of Buf* eau ac- 
- tivities, is timely and reliable assistance. The Federal 
contra** has allowed the PES'to enhance its ability to 
provide a variety of activities in a. meaningful manner 
to the Bureau of Vocational Rehabiltation. 

~* If you desire more information .on this topic, 
telephone- the .writer at 1717) 787-5123. 
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WELCOME 



Dear Program Participants, 



We welcome you! And anticipate tl^at this conference will provide a forum for all those in- 
terested in the futurej>toocational rehabilitation to reason together about evaluatioi^and all 

that it mean^. ' *' 

• " 1 ' 

The Model Evaluation Units (MEU) first national conference on evaluation will provide an 
environment for. (Jiscussion of concepts for better management through better use of 
dwindling resources. It is sponsored by the West Virginia Rehabilitation Research and 
Training Center, coordinating contractor fonthe Model Evaluatioti.Unit project, which is fund- 
ed by RSA for the purpose of developing s£ate VR program and policy systems through modtfT' 
evaluation/management information support urfits. - ' J 

The creation of the Model Evaluation Units provides a visible point of focus for .ad- 
ministrative needs as they relate to program evaluation. Of equil 'importance is 'the develop- 
ment of a coherent National agenda in program evaluation and tp exchange the instrumen- 
talities whereby program evaluation can be translated into actual practice. ' 7 

* ' * 

Staff from the six MEUs (Delaware Division of VR', Michigan VR Service, Mississippi VR for 
-the Blind, Oregon 4 VR Division, Pennsylvania Bureau of VR, and Virginia Department of 
Rehabilitative Services) will be on. hand to- impart technicaland experiential program evalua- 
tion information which they have translated into actual practice. Also available for assistance * 
will be representatives from CSAVR, RSA, and private contractors involyed irvthe MEU pro- 
ject. Other participants it is hoped will share their knowledge and experience in the discus- 
sions to follow in the next few days. 

The MEUs invite you to share with each of us here your ideas for better management 
through program evaluation. 
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THEME 



"Better Management Through Program Evaluation" 

What will you gain by attending? The conference participants will leave with: 

1. An ability to identify and prioritize for themselves the essential and ^desirable 
components of a comprehensive programed policy information support system; 1 

2. An appreciation of the fiscal and personnel resources required to implement components 
of aii information support system; 



appreciation of the importance of implementing a high quality, comprehensive 
^te/federal MIS system recognizing the requirements of the congressional and 
cutive branches; 



9. An appreciatipn of the potential in-house contributions of various information system 
components, including contributions to policy development, program accountability, 
political health, and budget preparation and presentation; 

* 

5. An ability to associate sources of information and technical assistance with perceived 
needs. 




PROGRAM 



? "The Management Uses Of Evaluation Information" 
National Conference, June 30 - July«2, 1981 



Tuesday, Jupe 30, 1981 



7:00 p.m. - 10:00 p.m. 




Welcoming "Reception 
Registration 



Reception Committee 



Poolside 



v. 



Guy Veach, M.A. Ed. 
Delaware Division of Vocational 
Rehabilitation 

Robert Struthers, Ph.D. 
• Michigan Rehabilitation Services 

Wendell Cox, M.S.- 
Mississippi Vocational Rehabilitation 
For the Blind 

Neil Sherwood, $L.S. 
Oregon Vocational Rehabilitation 
-^Division 

Harry Guise 
Pennsylvania Bureau of 
Vocational Rehabilitation 



W.H, Brownfield,>.S. 
Virginia, Program Evaluation 



, James Taylor, Ph.D. 
Rehabilitation Services Administration 

. Thomas G. Tyrrell 
Chief r Arizona Rehabilitation 

. Services Bureau . 

Nan E. BrenzeCEd.p., * 
Conference Chairperson 
West Virginia R & T 

Richard A. Nida, Ph.D. 
MEU Coordinator 
West Virginia R & T 



•ERIC 



463' 



308 



*8:00 a.m. - 9:00. a.m/ 
\ 

9:00 a.m, • 10:15 a.™. 



10:15 a.m. 
10:30 a.m. 



10:30 a.m. t 
12:00 Ngon^ 



' Wednesday, July 1, 1981 

* A 

: Registration. 

v Welcoming Remarks . 

Thomas G. Tyrrell 
Administrator * 
Arizona' Department of 

Economic Security 
Rehabilitation Services , 
Administration 

James Taylor, Ph.D. 
Project Officer 
Rehabilitation Services 
, Administration 

"Overview of the 
" Model Evaluation Unit 

(MEU) Concept" ~ \ 

Richard A. Nida, Ph.D. 
Project Manager 
West Virginia R & T Center 

. Coffee Break 

"Evaluation and 
.Accountability in the Future-* 
Implications of the 
Reagan Administration 1 ^" 

"A Federal Perspective" 

\ v Fr^drick Sachs 
Acting Dgputy^Commissioner 
^Rehabilitation Services 

.^Administration 

"A State Perspective" 
' Terry Etling" 
Director. Bureau of Planning! 
Evaluation & Interagency . 

Relations 
.Ohio Rehabilitation Services 
Commission' . * * 



Navajo Ballroom" 
Navajo Ballroom 



Navajo Ballroom 



30,9 . 
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12:00 Noon - 1:30 p.m. 
1:30 pirn. -.2:00 p.m. 



2:15 pjh^ 3:00 pan. 

v 

. V 




3:00 p.m. • 3:15 p,m. 
3:15 p.m. • 4:00 p.m. 



lunch 

"Status* 
Task 4> Testing of 
New Evaluation Standards 

Task 5 ^Testing of the 
Vocational Medical Facilities 
Importing System 
Rodney Pelton, Ph.D. 
Rehabilitation Services 
Administration >, 

s\ 

Panel Discussion 
{faith audience participation) 
"Role and Function of Program 
Evaluation in a State ^Agency 

3 AH This With Liniited Funds" 

♦ * 

,Neil. Sherwood, M.S. 
* Assistant Administrator for 
• _ * Administrative Support 
. Oregon Vocational * , 
/ ' Rehabilitation Division 

Guy Veacfr,M.A^Ed. 
Administrator of Planning and 
EvalualKn Uiijt 
Division bf*Vj*4tional 
Rehabilitation » 
State of Delaware 

Coffee Break 

Simultaneous presentations . 
by MEUs 



J "Caseload Assessment: 
Measuring Beyond 26Y' 
W.H. Bfownfield, MJS\ 
Director Prograirf Evaluation 
Principal Investigate MEU 
Depte of Rehab. Services 
Commonwealth of Virginia 



A 



310' 



5:00 p.m.; 



"Approaches to Case Review 
Activities in State VR 
Prp^ram Evaluation" 
DonaljiHossler, M.Ed., CRG 
Program Analyst 
Bureau of Vocational 
: ^^-Rehabilitation . . 



Pap ago Room, 



Pennslyvania 

"What Are All Those Numbers 
Uaqd For?. . .Evaluation 

instate VR Agency" 
Robert Struthers, Ph.D. 
Principal Investigator, MEU 
Michigan Rehabilitation 
Services* 

' Panel Discussion 
(With audience participation) 
"Program Evaluation As A 
Top Management Tool" 

J 

Arthur Lingle 
Management Analyst 
Bureau of Vocational ' * 
< Rehabilitation 
Pennsylvania * 

, : Hobert Struthers, PH.D. 
Principal Investigator, MEU 
. Michigan Rehabilitation _ 
SeSftices . > 



y 



Zuni Room 



Navajo Ballroom" 



9fi5*ajn. 



2T* 



Guy Veach; M.A.Ed! 
Administrator, of Pfenning & 
* - c Evaluation Unit 

.Divisional Vocational 
, Rehabilitation 

. State Of Delaware 

ifuirsday, July 2^19^1^ 



*Cok/PeneIit Model for. Use 
* ByState VR Program" 
*' ' 1 Ross Moran. Ph.D. 

'Researcher * • 
. v j^Orcgbn Vocational " 
fiehabiiitation Division" 



Navajo Ballroom 
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PR0GRAM CHANGES v g 



"pZease add. to -your program the following speaker 



for July 1, 4:15"- 5:0Q-p.M. 
V 



"Program Evaluation as a Top Management TooVJ 



Gabriel M<&kvsonn " - ' J 
Consultant for Planning T 

arzH "Evaluation Unit 
division of Vocational 

Rehabilitation 
State of Delaware 



j 



4S> 



1 
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9:15 a.m. ■ 10:00 a.m. 



"Relationship of PE To A 
Management Information 
System in a State VR Agency" 
• ArtLingje* 
Management Analyst 
Pennsylvania Bureau of 
Vocational Rehabilitation 



Navajo Ballroom^ 



Wendell Cox, M.S. 
Principal Investigator, MEU 

Mississippi Vocational 
Rehabilitation for the v Blind 

Alfred P. Grapoli 
Manager of Management 

Support Services 
Vocational Rehabilitation 
Division 
State of Oregon , 



,10:00 a.m. • 10:20 a.m. 



Coffee Breafc 



10:20 a.m. - 11:00 a.m. 



"Use of EDP in Providing 
For State Agency , 
Management Decision Making" 

Guy Veach, M.A. Ed. 
Administrator of Planning and 
Evaluation Unit 
Division of Vocational 
, Rehabilitation 
State of Delaware 



\ 



Navajo Ballroom 



* ft. 



r > „ a 
c 



Art Lingle* 
Management Analysts 
Bureau of Vocational 

Rehabilitation ' 
State of Pennsylvania 

' Robert Butlej 
Chief of Management 
. Information Services Unit 
\ rf, Division of Vocational t 

Rehabilitation 
' State of bfetfon 

■ •-• • "\ , 



H:10^.m. - 11:40 a.m. 



Don McLaughlin, M.A. 
Systems Analyst 
West Virginia R & T Center 

Conference Summary and 
"Hope of jthq Future for 
Program Evaluation" s . 
Joseph Moriarty, PhD. « 
Director 
West Virgifiia.R & T Center 



Navajo Ballrao'm 



11:50 a.m. - 1:00 p.m. 



Adjournment 

Informal Sessions for 
Conference Participants 
'Technical Assistance and 
Courtesy Consultation," 

Ralph Turner, Ph.D. 
ABT Associates, Inc. 

Mark Rogers, M.S. 
Berkeley Planning Associates ■ 

Model Evaluation Units 

Rodney Pelton, Ph.D. 
RSA Representative 

West Virginia R &*T Center 



, Zuni Room 



Navajo Ballroom L 



4>gt giM^Navajo Ballroom. II ' 
> 

Pap^go Room 



"Papago Room 



. / > 



« • ; * 



\ 



c 



> 



Planning Committee 
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Nan E. Breirzel, Ed. D. , - 
Research Associate » 
WVRTC '. \ 
Dunbar, West Virginia 

Wendell Cox, M.S. * 
Principal Investigator 
Voc. Rehab, for the Blind 
Jackson, Mississippi 

Arthur J. Dixon ' < 
x\ssistant Secretary 
Office of Human Development' 
Baton Rouge, Lpuisiana 

Richard A. Nida, Ph.D. 
* Project Manager 
MEU Coordinating Contract 
WVRTC . ' • 
Dunbar, West Virginia 



Rodney Pelton, PhD. 
Chief, Division of Evaluation, RJ3A 
v ^ashmgton, D.C. 

• . Neil Sherwood* M.S. 

Principal Investigator 
. • Voc. Rehab. Division 
Salem, Oregon 

Susan Stoddard, M.S. 
Berkeley Planning Associates 
Berkeley, California 



James Taylor, Ph.D. 
Project Officer 
MEU Contract 
U.S. Department of Education 
Washington, D.C. 



Special Appreciation 



The Model Evaluation Units express 'their deep and heartfelt appreciation to the Council of 
State Administrators of Vocational Rehabilitation its assistance and cooperation in planning 
for this meeting. ^ . * * " 

• * 
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Advisory Committee 



Representatives from U.S. Department of Education 
Office of Special Education and Rehabilitative Services 
Rehabilitation* Services Administration 



Wesley W. Geigel, Director 

Div ision of Rehabilitation Facilities 

Washington, D.C. 



Willman Massie, Coordinator 

Innovation and Expansion Grant* Program 

Washington, D.C. 



Lawrenc^I. Mars 
Statistician and Leader oi the Data 
• ^ Analysis Tea;n 

Washington D.C. 

Robert L. Davis, Commissioner 
RSA Region VI 
Dallas Texas 



Harold D. Viaille, PhD. 
RSA Region VI 
Dajlas, Texas 

\ ' ' 

, . Representatives from the Cc^pcil of State' Administrators 

* ■<* For Vocational Rehabilitation 



Frank Campbell, Ph.D. 

RSA Region X 
b Seattle, Washington 



John M. Cobun, Ph.D 
Assistant State Superintendent 
Vocational Rehabilitation 
Baltimore, Maryland" 

Terry Etling^Director 
CWeau of Planning & Evaluation 
And Interagency Relations 
Columbus, Ohio 



Robert W. Dingwall, EdIX 
Division of Vocational Rehabilitation,, 
« . Director 

Cheyenne, Wyoming 

" * 

Lowell E. Green 
v Administrative Assistant 

Oklahoma City, Oklahoma 



Representative .from Regional Rehabilitation Research Institutes 

r m 

* , . Don Harrison, Ph.D.' 

Associate Professor & Director * 
University of Michigan, RRRI • 
Ann Arbor, Michigan 

RepresentativC-from Research Utilization Laboratory * 
• . tCD Rehabilitation & Research Center 



\ 



^larvin^eisinger . 
Associate Supervisor 
New Xork, New Yor,k 



Advisory Committee 

* ; v Representative from Oklahoma Rehabilitation^ 
J Administration & Management Program 

: L.H. Antey 

Director 
Norman, Oklahoma 

* 

• Research*Utilization Specialist Representative 

Thomas E. Backer, Ph.D. ' \ 
Human Interaction' Research Institute 
Los Angeles, California* 

Consumer Representative 

\ Kathy Miller . 
University Center for^nternational Rehabilitation 
East Lansing, Michigan * V 

RSA Management Information SystSm Representative 

4 » 

Ron Geddes 
ABT Associates, Inc. 
^Cambridge, Massachusettes ^ 



Model Units Principal Investigators 

Guy Veach, M.A. Ed. 
^ Divisionof Vocational Rehabilitation 
v Department of Labor 
820 French Street 
Wilmington, DelaJwaje 19801 s 
y (302)571-2874 



Robert Struthers, Ph.D. 
Michigan Rehabilitation Servfce 
Michigan Department of Education 

P.O. Box 30010 . * 
.Lsansing, Michigan 48909 
('517)373-3970 

Wendell Cox, M.S. 
Mississippi Vocational Rehabilitation for the Blind 
P.O. Box 4872 
Jackson/Mississippi 39216 
(601)354-7704- 

Neil^herwood, M.S. 
Oregon Vocational Rehabilitation Division 
* Department of Human Resources 
2045 Silverton Road, N.E. 
Salem, Oregon 97310 
< (503)378-3830 

Harry Guise 
Bureau of Vocational Rehabilitation 
Department of Labor & Industry 
Commonwealth of Pennsylvania 
7th and Forster Streets 
% Harrisburg, Pennsylvania 17120 < 
" . (717)7^7-7312* 

W.H. Browpfield, M.S. 
Director, Program Evaluation 
Department of Rehabilitative Services 
Commonwealth of Virginia 
4901 Fitzhugh Avenue 
P.O. Box 11045 
Richmond/Virginia 23230 
* ' (804)257-0271 * 
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Cohnfmonwealth of Pennsylvania 
, * Department* of Labor and industry 

Pennsylvania ~ * 
Bureau of . 
Vqcational 
• Rehabilitation 



Labor and Industry Building - Seventh and Forster Streets 
Harnsburg, Pennsylvania 17120 
MODEL EVALUATION STATE 



Pennsylvania has, „is and will continue to benefit fr,om it partici- 
pation in the National Model Evaluation Unit Federal Contract with 
•the U.S. Department of Education. The Program Evaluation Section oP 
the Pennsylvania Bureau of Vocational. Rehabilitation began contract 
work activity in October, 1978, contract funding concludes in" , • 
September, 19S1. We have developed several products from a, variety 
of tasks that have helped us and should proye helpful to other 
socilil service agencies and personnel in the* field of program eval- 
uation. This handout describes briefly our contract „work . We. 
encourage ydu tocontact, us for detailed information/ 



Contract work activity has required technical assistance from a . 
variety of public and priva.te sources. Tracking and monitoring* sys- 
tems have been refined and several new dissemination and utilization 
strategies were utilized. The- results of the field testing of the ' 
Program Evaluation Standards, Project Standards' and a -Facilities 
Information System will be™ available .when the contract terminates 




Program Evaluation Sectjorf 



-i 



-MORE- 
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The Pennsylvania Bureau. of Vocational Rehabilitation's Program Evaluation Section 
is completing several mandated contract activities and integrating them into other 
developing and refined methodologies (products), some of which are listed briefly 
below: x 

MANAGEMENT INFORMATION SYSTEM - A process has been developed and utilised to 
review and assess Bureau usage of materials disseminated by the Program Evaluation 
Section at the central and district office levels. Additionally, an "Informational 
Request Procedure" to respond to central and district office and external needs . 
for information is operational* 

CASE REVIEW PROCESS - This distinctive procedure is concerned with case service 
patterns and provides information ^for program monitoring. The process has been, 
in existence for several years and" was refined during the contract period.. 
Examples of case reviews completed are: 1) "Suitability of the yocational. 
Objective", and . 2) "Clients Closed as Homemakers" .' 

CLIENT FOLLOW-UP STUDIES - Methods utilized to determine how well the rehabili- 
tation process is meeting client needs and in compliance with Federal Program 
Evaluation Standards and State policies. Additionally , \/e* are matching follow- 
up suryey information with the client master file record for purposes of 4 f 
validity. Cross tabulation of items is being conducted utilizing the Statistical 
Analysis System (SAS), a computer software package. 

PROFILgl&NALYSIS TECHNIQUE (PAT) - This objective assessment procedure is 
utilised r^Tcompare performance achievement for factors betjTfeerLJiqu^selors., 
district s-J»-and States. ' PAT exhibits program strengths and weaknesses *o loaanage- 
ment in terms of performance factors which are staru^ardized. 

< '• * 

"PILOT STUDY FOR COLLECTION OF SIMILAR BENEFITS - .This system collects and 
dociiments the use of similar benefits via electronic tf&ta processing. The pilot 
study began April 1, 1981, in one of our 15 district offices. The district 
offic^ is 'entering the following information':, 1) sources, 2) service provided, • 
and 3) dollars saved, utilizing the teleprocessing units located in the district 
office. Extensive training was completed of counselors, supervisors, admini- 
strators,,: and clerical* staff : in defining, documenting, and entering the infor- 
>^*mation to the compuete'r fil^s. 

TECHNICAL ASSISTANCE CENTER (TAC) - This information system respqnds to a 
variety, of requests from management and^staff in the -field and our central 
office. The information available, ranges from medical aspects of disability 
to principles of planning, programming evaluation and management. !l Th£ Rehab 
Brief", published by the University of Florida RRI, is our vehicle to, disseminate 
information" related to joiirnals, books and periQdicais available^ in our TAC. 

Although -funding for the contract .will conclude September 30, 1981, Pennsyl- 
vania will continue to refine and .develop many of the contract era work 
activities in program evaluation. If you desire more' information telephone 
or write to: ( * 

^> * " Harry W. Guise, Administrator 
' , Program Evaluation Section * 

Bureau <of Vocational Rehabilitation ' * 

% 13 1 7 Labor and Industry Building 
Seventh and Forster Streets 
* Harrisburg, Pennsylvania 17120 

(717) 787-5123 or (717) 787-7312 



• • .* • * APPEHDIX 32 

. • ' - ' . \ ^ ; ; " kpuJUL 23, 19 BO 

MEU ^MaftOp .x'. ' ' ' 

Ha/w# W. C<uAe, Adnwuit'tato* '- . ' 
Evaluation Station, . \\ "" 

/Ittwitton.: ' (OOUam W. JenUnA/ Ed.P. . \- - ' 9 

ReAeotc/t ?&ychologi*t - Root* 1 377 J 




ZLchaAd L. Cohen 
CaAe Sexviee Eydtuatpfi. 
Ktgion I *■ ^< 

V ' • ■ • ■ • ♦ 

s •.'/ .. . - • § 

BoAed orf otu dUmision ' skuung mj la&t Centnal p,(^cce v>ktt f tne. iolto.)ing • 
Za a dtatft p/u>poAat £o* an MEU Cfofe&ftop to - utilize, the. present budget *.tm . 
$o* t(UA pu^tpdAC 04 to copintt t/ic tfundA ioX uac <&jUn$-t!it next contract , 
</ea»t: • * ►•' * ' V 

A PROPOSAL FOR THEvCWui^^^frOP S> THE'PA MEU FTO%: EVALUATION 
SECnOU STAf-F OF OTffER STATE AGENCIES ON THE FOLLOWING EVALUATION SECTION' t 
ACTIVITIES:- . ' \ • \ " ' 

- ivaXiuvbipn Section 6.taU xetatlonhhipA with pthw pee* gioup, Centeat orf^ice. 

ami ^tetdAtajtf . . *\ • '. . , - • .in—*' 

.- State agenctf evaluation iectlon internal techucnl aMUtance acUvAXtei 
' - Opa/iotionA-and tfuncttonA^ a State ag>nte# *cAou/ice. inioxmation cent&X. 
' - EAtef&cAktng and opexatCng <£ Stags. vL.guve.tj lihxaxti. * , 

• •* ' « . **. . ■ ' 

The. tatoet'dudhflCJi l& State, agency AtaM tSiai axe ai Argued MJizet functional 
xeApbn&ibiUty (,ox ann on alt' of, .tin activitiM enuintxatcd .above. The. 
pxintipat chkna&teJd&Uc. oi the '.toxlishop i* one oi, by and iox the. agency 
. it&ii that one. tfie implementox&.. , « N \ . 

Vximaxu Ta*ge£ G^ioup - 'SiAte^i and iriedtum-Atze,d State agenda 
* ''Se.conaa.'uj Tnaget Gxoup - Remaoung State agenc^A 

'Gnoup Uze. v , 4 -f '.^^25 * • • 

P^og^am OtycuUzation and Content: ' . . , ^ 

Fotrt 3-^ fiotvt unttA pxutrvted oveAAvo Hull dat/A tomtit ona day pntczding and 
lottowing ion tnavel. Each umitnvlU, deal tilth one oi the topical axeab 
pfveviou^ty lUte.d. . ° : . . : ' 

Tfte wo^feA^iop AteL' membetA i^t« bp selected and <oKganized item the pfieieyiXly 
OAAtflned- a taU 'ol "txe Evaluation Section and Ml be tompohe(Lob leven 7) 
p/UrtfcaJteA. *T/tc u)04feA/io^,dacctoA f adm^tiattuc ,coo-ytnato^ f <ouA [4] wU 
PnuefceM/wohdlnatoM, one fax eacix topical toiea and <a nzeoKdex wponstbU . 
to. keep a xe'eotut oi the pKoceedLingi and pxepaxe a poAt wotik&hop handbook. 
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MEU Workshop 
April 23, 19 tO 
Page 2 



T ^ < L UW ^ t P******** to responsible {or.the'planninq, preparation and presenting 
o& his assigned topic. He may > itiAitii tiie availability of funds, utilize v 
outside peel specialists for portions of the ante* 4 v 

, %ach unit utUl be pres'oited in sequence, and alt participants vtitt attend 
each unit. ■ Pfitezntattons i\silt stress maximim participation by utilizing 
tltiier fiUl-or^smalLgroup-activity.~ ' . 

Location: The nost suitable and available. conference worn in the LaboK and * 
industry Building. 

Pus. Workshop Inquiry and Survey: 

During toorkshpp planning an inquiry-surveij letter uxUl^.bz sent, to the target, 
group State agencii sections outlining the. proposed program, tintaM^C dates 
utcatLon and stipend. Agencies will be lequested to respond indicating* - , ■ 
individual* oi the number interested in attendance.' and suggestions regarding " 
plagium content. *** w , • 

Budget* $ 6,00V is presently available in the MEU budget for this purpose. >* * 
A possible. apportion of those funds could provide: '.. * ' 

. „ • v • * • v 

Outside Specialist Presenters 3 ' $ ,SOO t p.O . 

Miscellaneous Admittist^ative-^xpense - , ■ Z^SU'OO '■' 

Keinbursement to Participants $5,000 I 25 * $200*; 5,000..00 

* - ■ V $6,000.00 ' 

•77t*ee nights lodging atid subsistence 

If it is desired that toe proceed uiith this proje-ct, the appointment of, the 
workshop staff Mould be Hie'next step followed by an oKganization meeting 
to. confirm, modify or change- elements oi this proposal and to *as*inn* specific 
■ planning tasks. 1 ** . .. * . ' . » 
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_ BUREAU 0F\^CAT|pNAU>REHA8!LITATI0N 



0«pirtmint ol If bo/ and Industry 

* 



labor and Industry Building 

\ • - 

Seventh and Forster Streets^ , ^ , 
Hamburg, Pennsylvania 17120 

September 29, 1980 
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% f ' * 

The Pennsylvania Bureau of Vocational Rehabilitation has completed 2 years of a three 
year Federal Rehibilitation Administration contract to develop a* Model Evaluation Unit.* 
This contract is entitled "Model Evaluation Management Information Support System. " 

Our contractural activities include ffche following: 

1. -Refinement of a functi^al program^^vdluation unit within our agency which in- 
cludes addressing sectional organization, personnel resources and available data £ 
for effective managerial utilization in program planning and decision making. 

2. Development of a Management Infprmation System wathin our agency that ensures 
. appropriate ^audience utilization needs, < ' ' M 

s . • - o 

3. Develop new-methods and techniques for evaluating, such as utilization ot * 
'Similar Benefits and Facilities Cost.and No Cost Data Base/ 

4. Deyelcpcuwt of, a Program Evaluation Section plan for effective* external and 
internal dissemination and utilization of information. • * 

5«- Development of; a comprehensive program-evaluation plan for PA BVR Agency. 

6. Field test prdposed Federal Standards for program evaluation. ^ 

* Through our contract activities, we have acquired information and knowledge that i&ay 

be of interest and value to states similar to ours; consequently, we are planning to dis 

% y -seminate information to you in a fopn to be determined, based upon your responses to the 
Si * attached questionnaire. x \ 

* V Please consult with yotfr *pi*>gram evaluation personnel' incompleting the enclosed 1 ' , 

t attachment.- if you have any questions, please call my office at (717)'787-'5244 • An en- 
closed envelope'has been proyidgd'for your responses. Thank you for your time and comments 
in this endeavor. . . ' ^ • y > * 



1S- 



Sincerely, 



Clark D. Kennedy 
Acting Director k 



v 
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Thi-3 checkli*^j^l£^ you desire, more information in order to 

develop or refine afeai'b^ybur a^eScj*}'^ . % 

Please list numerical magnitude of interest i^Vth^^clO ^itf iiHVttlef tf of the statement* 

/~7 b. weighted case closure system ^ t \n > 

r*77 c. Federal Standards^ 6 9 survey development, implementation and analysis . 

/~7 e. facilities 'information system development .a34;#SHA$at^Sfe % 

r~l fl new federal program evaluation standards-- ^..^v^,, 

/T7 g. organizational development of a VR program 'evaluation 1 are*C£ion 

; * * * * * 

/*""*7 * h. prpfile analysis technique 

. I~7 irf budget control and accountability of a federal con£r^efe>(\j, *vy ' 

■ - -- — — * * 

£H7 j* management of <a federal contract 

k. management information system development as /elated to a VR program evaluation 

* section Si TA " rT . 

^ jT g 1. utilization o£ EDP within a VR program evaluation section 

* f ) m. development and implementation of a case review process on VR program" evaluation 

r~7 n, role and functions of B W^Si^^Aeti ^i^ ^^J^y^^itpn ' 4 - 



: 7 V^J^lp - 1 ^^^''; ****** f^Brlfeg Address 

y - * 



. . C. 



Title i - Office Phone 

• * 1 " 1 v — r 

P1*ase return in the «aclosed envelope. Thank, you. % , < * ' 
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, * . October l, 19 SO 9 

TENTATIVE CONFERENCE DETAILS 
STAFF PLANNING QUIPANCE 


1 




CONFERENCE OBJECTIVE . ' ' * 






fttouide to zounttKpaAt 6tn.&& {from $UteA Statu VR Agencies few*u<*edge and 
expedience (joined doting i/te Mode£ Evaluation Unit Contract pesuod. The. con- t 
ItAtnct pJioqnam uxiLL emphasizz products and proctitis generated by the £o (touting 
tont/iact taAki : Modet Evaluation Unit- Organization and Functioning f Task 2; v 
Vtvttopnavt o] New Evaluation tepacUtLes, Task 6; and ZnteAnal VisseiiUnation N . 
and Utilization, T<uk 7. 


n 


J 


Provide a contfeAence Structure and £orihm tp ,£aciJUtate the &ree exc/iange 
otf fcnotcledge and idea* between Program Evaluation Unit sta££ membeM o£ the 
attending states. ' 




t 

n. 


TARGET AUDIENCE 

* t * * 4 




in. 


LOCATION ^ ' • U- 


• 








17. 


TIME AW PATES • * 


:\ 


r 

• 


• > * * * • * 

• > - 




< 


CONFERENCE ORGANIZATION AND CONTENT * 

(He contfe£fcn<je promotional afid inqutrtf activities to soluHi target 
audience participation and to assist In Emulating topical arexis otf interest . 
to p^ttcipants . • * 


• 

\ 

\ 




T/ie contfeAence tuitd 6e conducted oveA -fwo tfu£fc calendar, day* allowing the • ' 
preceding and {ottocaing dat/a £o/i -i*ave£. T#e p*£4en£x£ion otf eacA topical arm 
mill utilize methods that lactiitatt maoumum participation. ' 

Tht , follouting topical areas arz,to provide a basis {or initial planning: 
it * * * * • 


f 


9f. 


SESPiWSISILITIES * * 


( 










jCOORDi'nATTON WITH OUTSWE RESOURCES <^ 






t • • * • * • 




VIII. 


v CONFERENCE EVALUATION _ # * • 


* 

-A 




A 4ucfe6£e i)i6^uwient mea4UA.c thz cm^ctence e^ectioene44 tactt 6e 
denned and utilized. +> 

• 




• 


. ^ ." ' . 481 "\- r 

1 ' • ... 


;;• Bye 


* 
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■ * SISTER mTES CONFERENCE OaXobzAll, 19 SO 

OigayvLzation Mectoig Ag tnda 

1 . Con^Jkm taAgzt aixdiwcz 

. *, 
A. ?Kim<vuj 

* 

8. Szconda/iy 

2. A&tign and dz^ini A.t6pott6ib£LLtU& x 
A. Con^zAznct Viwcton. 

8> PtogAom CaafuLLnaAoK 

C. AdminibtAaCivt CooKdJbwuton. » 

3. E&tablUh twtcutLvz prwgtuxm - v * . 
A. Sex topicaJL oacm in pfUonJjO^ - ^ 

. 3D 

8. A44-cgn KteponAibilcUqA £ok zanh 

4. AdminiitvUive. doZlviti^ . tJ . 
A. E&tabtUk daZte and tbneA * 

8. Ettabtith pioquxm faciZLUeA need* Tm^ga^ - External . ' 

. C* Review budget. . % , > * 

SiS^^' COn ^ ence " ^ ioUo^'tetf aULvm^ rm&t be 

Wee/a F/iom Cor.j{eAence ^ 

A. FKomotianaZ - /teapon-m £tom punaAy tafiptf fuutuncz. ^ 

— : 8 - ° Emotional - iztondanjj audience, attendance, conned - 

C. , ?iog/uzm completed and bfl.odiuA.e~ pUntkd - jt - 

V. External &a$UsittiA conltiuned * 

, ,< OvuUJjdz pieAwteM confirmed 

Attendee adminl6t/iatCvz deXaXZ& 
Utg^tAatLon] pn.zpaA.td, hent and KefrJtned K > 



F. 
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Topic Title 

Other Related Topics 

Presentation Objective 

Presentation Method 

• Preserver Team 
» 

Internal - Central Office 

V 

Internal - Field 
External -'Consultants 

External - Other 

\ „ 

Audio-Visual Needs 
Handouts (Printed Material) 



Facility Requiremen ts 

*- * * * 

g 

Demonstration(s) , Tour ■ 
* 

Outline af Presentation and Time Requirements 



. SISfER STATES COWFtRPWCE 
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CO 
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£ s f 

* TOPI* - * 


PRcScW/tK 3 


. OBJECTIVE 
TdVW EMPHASIS 


PLAWNIWG GUIPANCE 

PURPOSE TO: 


METHOD . 

— A 


HCWt * 


WHAT? 1 


GENERAL- 1 " ei(T 
INFORMATION |. mm 


INSTRUCT 




CASE REVIEW PROCESS 


HARRY GUISE 


75 * 


25 








PANEL 




INTRODUCTION & AGENCY 
ORGANIZATION 


* 

CLARK KENNEDY \ 
. HARRY GUISE 


• 


100 


« 


\ 

-v 1 


• 

• 


LECTURE 




T.A.C. t - 


LEAH KUHNS 


5° 


50 


- 




• <* 

n 






EDP UTILIZATION . 


'a^t ltnglb 


'* 80 


2Q 


ft 


' ■ 




PANEL 




PAT 

% - * 

• 


PILL SCHILDT * 

i ■ f 


50 


50 






> 




• 


. \ _ 

SIST^R-StATES " ' 
PRESENTATIONS *'\ 


k • 

HARRY GUISE 




100 

> 


^ 






■ L 




4 OTHER STATES PRESENT \ 
PROCESS /-PRODUCTS \ 


• * ■ 


" RESERVE 


2 HOURS ' 


1 


* 


r » 






.SIMLA* 'BENEFITS 


ANDY CHOPAK * 


80 


20 




1 " 

■ *A — 


\ 


PANEL 


; 


P.B. STANDARDS 


lJ - ^ 

1 • ANDY CHOPAK ' 1 


75 . 


25 






• 






• 


ART LINGLS 

V %. 




> 






\ 

ft 


<- 

LECTURE 



45G 



T 



SISTER STATES CONFERENCE 





TOPIC 


PRESENTER ? 


08JKTII/E PLANNING GUIDANCE 




Tone 


.EMPHASIS 


PURPOSE TO: 


MET^OO 

\'-' 


PROCESS \ 
HOWT <. 1 


»EWT t 
WHATf ' * 


' GENERAL 
IWFORMATIOW 


ORIEWT 


INSTRUCT* 




STANDARDS 6 t 9/U S. 


DANIEL GRIB 


75 




• - 


- 




LECTURE 


t 


BUDGE r. CONTKOL 


ill n/"\T rv CPTT7DD 

HAROLD SEITZER 


100 


*✓ 


« 






LECTURE 




cwc * r 


DAMTFL GRTR 


100 


1 


1 


• 








t 






> 














A. 

c 






y 






• 

■ ' « 






> 








o 

V 








• • a ; - 


s * 








- 






• 
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R. Cohen 
11-20-1980 


( 


SISTER STATE? CONFERENCE 
SUGGESTED READINGS 

IDEAS FOR TRAINING MANAGERS AND SUPERVISORS 
SUESSMUTH * , 900 

' PART 1 Section 1 Objectives . . 




» 




PART 2 - Section 11 & 12 Small Groups 


f 




% \ 

\ 


PART 3 Section 18 Questions 
Part 4 Section A - V 

EFFECTIVE PRESENTATION . 
. WHEELWRIGHT (SELF-TEACHING* COURSE) 

UNIT VI Outlining, & Data Gathering 
UNIT IX Preparing^ to Face t)>e Audience 




1 

• • • 
• * 


* 

1 


UNIT X At the Podium 
THE CONFERENCE BOOK ' 

NADLER & NADLER ~ 750 
(Good -reference on Conference Administration) 

OTHER GENERAL REFERENCES 

MAKING MEETINGS WORK „„■ • ■ 
BRADFORD 




• 




CONFERENCE PLANNING tch 
B^RkE & BECKHARD " 






* 


TAKING YOUR_MEETINGS OUT OF THE DOLDRUMS * 7 c n 

sTOdtek; kkium, umn & cole — , 750 

• 

** 

N '* - > ' ' ' 


« 


* 




* 




■I 


i 

1 
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COMMONWEALTH OF PENNSYLVANIA 

January .8, .1981 



subject} .Sister-States Conference 



• 1 



TO: 



P. E. Section Staff 
Richard Cohen 



jFROM: 



Donald E. 



Horsier Qf If 



" ^ is T .. is an "P date ° n the.developing 'conference. We will have our meeting 

r^ a T ry , f th ^ y ° U Sh0Uld have y° ur Pl-nning sheets completed and 
ready for discussion. . 

^ * I ' < • . . '• ' ' 

We have received all seven States questionnaires (primary audience) and by 

January 19th I will have personally called each state.. I have caUed four 

l t6S t0 T d ; te and "n^owd their ^attendance ; need for special lodging, 

EtlL£ «FV ?eX>S ° n8 C ° min ' 8 P relimin «y information on how' we plan 
to handle fiscal arrangements. . K 

o'f ' a e P r P v?ce SntrlLf "t^' b - reakS aW airline fleets via Purchase 

Janua^ 9th It J n ^''T" t0 - receiv ^- the Sheraton's, signed by them, by . 
January 9th. It will- then be ready t^o submit -to the 17th floor. 

In reviewing the "Final" state sheet, you 'will note an average score Of 1 or 2 
indicates good interest. It.appeajrs we could try to have several -mini 'presenta- 
tions of those that did<no£ tcore that well. At our Reading Meeting Dick and I 
plotted time schedules into blocks. We can discuss this mofe full? n J^ry " 
27th, For now, perhaps you could- think over alternatives to presenting iust • 
one subject for a 1 hour & 30 minute period; Presenting just 

seVme ^j^u^t ^ ?lannin ^ 8hMt f ° r Jamui1 * 27 meetin S' make *>M ' 

see ne. I believe item* are' moving along very well for preliminary planning of 

Z lljlltTl- me! m ° re inf ° rmati0n ° r ^suggestions,^ on ove/ 



ft 



( 
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HffiH IilTFRFST . 



Similar Behefits. 1.4 
Utilization of EDP 1.7 
MIS 1.7 

Organization PE Section 1/7 

Case Review 1.8 

New PE-Standards 1.8 



Sister States Conference 

Topic Selection 

January 27, 1081 \ ' 



HF-DIUH ItJTEREST 
FIS 2.0 

Weighted Case Closure^ 

Utilization *Full Time 
Case Reviewers 

PAT -2 . 1 



2.0 



2*0 




4 7 



NUMERICAL LISTINGS ARE 'AVERAGE OF ALL STATE RETURNS 



LOW HffEREST 

MGMT . Fed. Cbntract 
V.R. Library 2.4 
F.ed. 65& Survey Dev, 
•Budget Control 2.5 



2.4 



2.5 



3- 

CO 
CO 



erJc • 
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January 27, 1981 



TOPJCS SCHEDULE DEVELOPED AT THE yJAN . 27 
SISTER-STATES PLANNING MEETING 



. NEXT MEETING - February 24', 1981 - BE THERE 



Wesnesday - Sheraton Building 



TIME 



8:30 - 9:00 



ACTIVITY » 



RESPONSIBLE ' 
PERSON 



Introduction HWG 



Thursday * Labor & Industry Building 



ACTIVITY 



RESPONSIBLE 
PERSON 



Travel 



9:00 - 9:30 



Organization CK 



Facilities Inf. 
System CFIS) 



Art 



9': 30 -10:00 



States Presen- 
tations DH 



10:00-10:30 



.Guests 



DH 



Budget , 



Harold 



10:30-11:00 ' 



Break 



ireak 



11:00-11:30 



s: 



■ Bill 6 . 



11:30-12:00 



AC 



12:00-12*30 



Std, 6 & 9 



AC 



12:30-1:30 



LUNCH 



LUNCH 

T.AX. Leah / CAP Don 



1:30-2:00 



2:0ft-2:3O. 



V 

E n E 



CAB Don / ♦> T,A.C, Leah 
EDP Art 



2:30-3:00 



3:00-3:30 



3:30-4:30 



4:30-5:00 



Break, 



Similar Benefits AC 
Similar Benefits 



Break 



EDP 



Art 



nation 



HWG 
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•sisit* States c<mer£vce, Vkograai planning meting - . 

\ - 

Januatoj '11, J9$J ^ , 

AGENDA • UWUTES 

II. lack PE &ta&& mzmbzn dzAcrtihz topic pftz&zntatlon ioK zdch a+tlgn- 
mznt, &ivz to tin muxutte peA topic. Covza pxz&.zntatlon io/imat, 
timi KzquOLZmzntA and vlem Kzganding novo will wc can pnz&znt -thz 90 
topic, 1^ hzlp&ul, uaz pKZttntation planning tooKkAnzzt a* a guidz. 

III. Review *£4ut& o ppanXlclpantA 9 InqaOiy and thzln. p\z&znjznczh . m 20 



?V. EttablUk a tzntatlvz lUt.oi ouA pwgiam topic* and tbnz Kiqulnid 
(jo* tack bated on thz preceding &tzp&. 



336 c 



30 



V. Vzvzlop tliz two-day ptwgKam tchzdulz. by topic, timz KiqtuAid cuid 40 
sequence. Adjust and modify at nzcz£&a/iy. . 

' i \ 

l/I.' Review and! ok zttablUk timz goalt> £ok pKZtzntcution KzkzaMalb . JO 

VII. P£an nzxX mzzting on adirtij/iut/tjatlvz dztaltt*. 5 
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.1 V- « 



V* 



:00 - 9:30 



r30 - 10:00 



:C0 - 10:30 



» jo • moo 



^00 -J1:30 



:30 - J2:00 



:0O - 12:30 



:3a - J: 00 



rOO - .J<30 



:30 - 2:00 



:00 - i:30 



i30 - 3:00 



:0O - 3:30 



*30-- 4:00 



irffl - 4:30 



itSO - 5:00 



FIRST VAV, VKteQ'yi 



SISTER STATESCCONFERENCE - VETAlLEO PROGRAM PLANNING 



LOCATION 



1 "pgS^T 



SECOND PAY, VATE 



... jL 



. LOCATION 



V 



SISTER STATES CONFERENCE 
Planning Meeting 
February 24, 1981 

AGENDA 



1, Review Program Schedule and Assignments - Revise as Appropriate. 

* * * * ' 

2. Discussion Of Administrative Details 

A, Participants Lisf^nd Number 
» j* B • Advance „ Material 



C. 
D. 



E. 



s Content ami when sent 
Review Lodging. Arrangements 
Review Transportation Arrangements 
'To Harrisburg and return 
Local . \ 
Subsistence 



F. Financial Reimbursement Procedures 



3. Program Support Activities 

A* Handouts 

B. Participant Binder 

C. Visual Aids 

D. Recording^ Proceedings 

E. Facilities 

F. Breaks 



*4. Program Presentation Rehersals 

A. Purpose *" 

B. How Conducted 

C. Schedule 



ERIC: * 



.437 
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COMMONWEALTH OF PENNSYLVANIA 

February 24, 1981 



suexcTs, Siiter-States Conference Meeting 2-24-1981 



T0 » - All P.B. Staff 
"«om. Donald E. Hossler 



Following is a tunning narrative of the meeting. . -Major points conclude 'this piece of dissemi- 
nation* 

Don Hossler-will call Holiday Grantville and Sheraton-West to determine if their, prices are 
lover that Sheraton-East . If the rates are lower Don will-do^bid letters in preparation for 
a kickback from 17. Harry has targeted March 16 as the % d ate to check with Mr. Malloy on 
"*,"* ° £ ° ur cont ract for rooms from the Sheretpn-East . . Harry will meet with, us on March 23 
about whether to cancel or whether to move the conference back. Andy will include 6 & 9 in 
his evaluation standards presentation. Harold will handfe introductions of personalities and 
monitor time. Harry said a letter needs to be written to Kaye about need to participate in 
the conference. It will then be sent to Pacinelli. We expect about 25-30 participants in the 
conference. Zelda is presently typing a fcaster dist of who will be attending from each state - 
•if you want to make calls- to them. Advance material should include a basic welcome letter 
with costs to each individual state sent out about the first week of April. An attachment to " 
*his letter should include program, brochures about the- area, our ortanization chart, forms to 
oe completed, limousine schedule. Any advance program material is responsibility of the pre- 
senter. Don Hossler will check with Monty on doing a brochure for us. . 

^/JiV^ P" tici P ants to aifi from the Labor 4 Industry Building on May 21. Harry will 
ItU? ?™ sblnskl «b.out subsistence a\d financial reimbursement procedures. Don Hossler will 
fcTThTS Ull X u° n av ?^ lablll ty>of 3-hng binders; meanwhile Andy Wl continue getting folder, 
for the states, hopefully before April- 1. All presenters should have their information typed 

^o° P y"^^ 00 8 * X 11 Paper 30 iC fits in a rin * binder and is Pre-pundhed. This should be 
accomplished no later than early May. Some material such as "interest" stuff could be passed . 
outiater. Andy suggested each state should send us their organizational chart. Leah will b« 
consulted on the be sr method to' record proceeding* in concise -manner to be published in final * 

n! P °^' /\? e f d t0 g6t the 16tn £loor f0r "V*". Harry will see Al Frank. Harry, Bill, 
Harold and Dick offered to pick up the breaks on May 21. Leah has offered her coffee maker. 
Keuearsals should be presented with as many section personnel as possible. Leatrhas a critique 
sheet we can use. please note rehearsal dates and plan to attend. Leah should make copies of * 
critique sheet for each session. Rehearsals are 10:00/A.M. on 16th floor April 7, 14, 2l 4,28. 
Harry will make sure we can get them. 4 

A P r ^ 7 " 10:00 A.M. - Organization HWG, presentations by state - DEH, 

S*M ^t" 418 ' AC * A ? ril 14 " J 0500 A ' M - CRS " HWG ' Si ™ l <" Benefits - AC, ApriUl - 

eJ;SL A ion*" hJg ' " S '„ TAC - LPIC ' FIS F A ?- A P ril 28 " ™ - »*». »P - AL, Closi^^T" 
'" „"" n " HWG * fach Presenter should give Don Hossler the correct title of his/her presen- 
tly - all day! " V ° ry lnvit ^ in 8- tee gingerbread man now has h brioe beer Tuesday and Thurs- 



Sea new corrected*program schedule (attached). 
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ACTION ITEMS ' ' 

OH - call local hotels and possibly developed letters; develop welcome letter and. attach- 
ment*; check with Monty^on doing brochures and 3-ro.ngr binders. 

aw?- see Dick Malloy 3-16; meet w/us 3-23 for ■ decision; write letter to Kaye^to be sent to 
PaTinelli; see Ed Krushinski about subsistence and financial reimbursement- , f 

AC - get folders printed ASAP* , ' , 

EVERYONE - get your presentations material typed appropriately in early May- Attend' " hea "als 
bring A-V stuff to rehearsals; give Don<Hossler correct title of your presentation before March 
6, 1981- * - ' \ ' % 

LK - consult w/others on best way to record proceedings get coffee maker for conference ; .get 
• critique sheet ready for rehearsals*. 
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QA-SOt 12-67 



COMMONWEALTH OF PENNSYLVANIA 



March 9, 1981/ 



subject:,. PA Sister-States Conference Planning 



to: MEU States, WV RTC \ 



trom: Harry Guise $ 
f Evaluation Section 

PA BVR MEU 




' JutTon"^ Zlll! C0 ° ference «d m are aiming for it to take, 

' be ^r t i°^." , " t * 1 ' 0rM ""^ *5« Pi-ini and MJ ' 

If you have questions feel free to telephone me.or speak with Don Hossler. * 
Phone 717-787-6695- • ' * - V 



attachments 
HWG : DEH : zmp 
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Dr. Gerlene Ross, Bureau Chief 
of Research and Innovation 
♦Room 1924, 99 V/ashington, Ave . 
Albany, NY 12234 ' 



MASTER LIST OF THOSE INDICATING ATTENDANCE 

.AT 

SEPTEMBER, 1981 SISTER-STATES CONFERENCE 
(518) 474-39^9 



,AdamR. Zawada, Adminis. 
Planning S Development Coor. 
Office af Voc Rehab 
1317 Winewpod Blvd." 
Tallahassee, FL 32301/ 

Joseph G. Taylor ,* Planner/Evaluator 
Same As Above 



(904) 488-5136 



Thorn Dederer, Manager 
Program Planning, J Eval . 
Grants 8 Contracts Div; 
623 Sast Adams 
P.O. Box 1587 
Sprajigfield, IL 62705 

Bill A. Forney, Super. 
Evaluation Unit ' 
Same As Above. 



\ 



(217) 782-9432 



Dennis Wysocki, 
Program Evaluator 
Ohio jtehab Services Comm. 
4656 Heaton Road 
(blumbus, OH 43229 ' 

Greg Shaw, Research 

Specialist 

Same As Above 



(614) 438-1302 



Paul F,. C. Mueller, Ph.D. 
Chief, .Program Evaluation 6 
Statistics Section 
CA State Dept. of Rehab/ 
Program Evaluation 6 Statistics 
830 K Street Mall 
Sacramento, CA 95814 

Mr- Robert H.- Philbeck 

Deputy Directdr , 

NC Division of Voc' Rehab Svs. ' 

P.O. Box 26053 

Raleigh, NC 2763,1 

Joe Marrow, 'Chief 
Evaluation S Program Review ; 
Same As Above 



(916) 445-9692 



Edward J. Flanders 
Director of Program Audit 
118 East Riverside Dr. 
Austin, TX 78704 

John Wylie 

Director "Program Operations 
Same As Above 



(919) 733-5766 



(512). 447-0186 



(512) 447-0305 
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RIC 



Commonwealth of Pennsylvania 
Department of Labor and Industry 



t.. 



Pennsylvania 
Bureau of 
Vocational 
Rehabilitation 



Labor and industry Building - Seventh and Forster Streets 
Harrisburg, Pennsylvania 17120 




Program Evaluation Section 
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Information Requirement Analysis** 
(Simplified) 

» 

We need a list and example of all information provided tg the Federal 
and State government from each Unit in the Bureau. 

In addition, we need an example of all source documents used to 
conpile this information. 

J> 

Me need a list of new % informational requi.cr.ents not currently available 

The steps to be taken on the existing informational requirements are 
as folTow: * , 

1. Hake a determination, in Conjunction with the Unjt concerned, as , 
■ to the disposition of the requirement. - , ^ 

A. Should Evaluation be Jiandling it in "its entirety. 

B. Should Evaluation Assist in its completion. 

C. Should Evaluation act as a consultant on its completion. 

2. " Effect a change if necessary- 

Note: Ail the above* in 1A^ IB, 1C and 2. should be included in the 
- Final Conceptual Framework. 

The steps to be taken on the new informational requirements are as folio 

.1. Identify the data elements and their sources in the proposed infor- 
mational requirements. 

2. Determine what outputs are desired by management and how to target 
then for the most practical usefulness* 

'3. Determine what manipulation will occur against raw data to develop' 
the information in as concise ajid useful for decision making as 
possible. 
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Report Analysis Procedure 
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Report Review Methodology 

The following subject areas : will be discussed -and outlined by the 
Report Review Committee prior to meeting with, a Report Recipient; 

1. The position of the recipient in conjunction to the information ' 
contained on~the report. 

2. The Evaluation Unit's opinion on the report's usefulness to the 
recipient. , 

* 

3. Dissemination of the report by the recipient. " • 

4. ' Other reports, not already reviewed, that would aid in the 

recipient's decision making process. 

5. Is the report functional to the recipient, or just generally 
informational? . • 

Based on the above discussion, the Committee will have a general pool 
of knowledge and will better be able to •conduct a fruitful meeting • 
with the recipient. They might also decide to not- schedule the meeting 
based on "these preliminary findings. «, 
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, , " APPENDIX 37 
REVIEW OF THE DISSEMINATION AND UTILIZATION OF REPORTS 

STEP 1. Develop Method for Review. "The methodology for reviewing the reports 
will take into consideration the position of the recipient of the report, the 
Evaluation Unit's opinion of t>he uses of the' report, etc. The methodology for" 
review will te outlined as a separate function. 



STEP 2. 



Determine which reports are to be reviewed. Certain reports are Federally 
mandated and certain ones are distributed as a courtesy to the recipient. The above 
consideration as well as others will be taken into account in determining yhich 
reports are to be reviewed. 

STEP 3. Using the cyrrent distribution chart, we will isolate, based on the 
Evaluation Unit expertise, those persons to be contacted 
*■ 

» * * 
STEP 4^ A meeting will be arranged witl) the selected persons to review the report 
We realize that 'at this time the recipient person involved may express disinterest, 
therefore. ending the .review process. This will affect our decision on whether to 
continue the distribution of the report to that recipient person', 

STEP 5. An explanation of the report, focused towards the recipient's area of 
expertise, will be presented along with possible uses. 

* ~ < v 

STEP 6. During, or shortly after the explanation, the recipient will -be asked if 
he actually needs the report. 

STEP 7 J Feedbacks will be gathered as to possible changes in the report to better 
suit the recipients' needs. 

STEP 8- A review of all contacts for a given report will be held with key 
Evaluation staff. 

STEP 9. -Results of the review, such as" changes in reports or new distribution"" 
will be put into effect as soon as possible subject to proper approval, 
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APPENDIX £8 

/ ^ 

Management Information Review Procedure - 

Objective 

This procedure is designed* to guide in the analysis and review of the 
current BVR Ilanagement Information System. 

Scone 

the initial analysis undertaken will constitute the State reports manually 
prepared by the Evaluation Unit and disseminated throughout the Bureau. 

Procedure * _ 

I. Plan of Action 4 

A. The 'Evaluation Staff will meet and discuss each report and its 
potential utilization by each recipient as it is currently 
structured and any changes or additions that might be tailored 
to 'the person T s particular needs. 

B. Set up a meeting, and beforehand give the person- a copy *of the 
reports and a^ritten or verbal agenda (all reports or pertinent 

m ones). ^ 

C. Meet with the person a*id discuss all $hi areas as outlined in 
II. Give a background 'on grant dnd MIS. 

^D. Meet -and ditecuss all the relevant ■ information obtained at the 
above raeejting with the Evaluation Staff. 

E. develop a plan for implementation of any changes, additions or 
deletions. This plan will include interaction with the user^ 
and ultimately, require, depending on the report, administrative 
approval . 

y F. Implement the pl'an as developed in E. 4 * t 

G. ^Conduct a follow-up meeting with user to discuss problem areas 
requiring further change and impact of the plan* 

II. Information to l»e collected. 

A. How do they use the report? 

B. -How often do. they use it? 

C. Itt^as on changes. ~ , 

* D. Can we summarize data for them*! 

E. vfliat about Ilanagement by Exception? 

t 

-F. Do they ^disseminate the report? 

51 1) • 
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* G. Do they want us to train then toSread it? 

H* Do they, use comparison of 4 data J f rom one year* .to the next? 



I. Do they use or could they usejbn6 report in cojvjunction with another? 
III., Things to keep in mind. 

A. Pass on information 1 obtained from person to person. _ 
c ' B. Might rre.ed another report to supplement currently received. 

* C. If -nothing else, this meeting will faniliarize the person v;ith the 

rapo % rt and our interest . ■ u N 

* v *~* * 

D.- Don't discount any changes that are not tailorable to existing 
reports. We can add new r r§ports. 
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Run tto. 



VKo*1230P 
VR612A5X 
VR61245P 
VR61260P, 
VR6U45P 
VR61380P 



VR613S0D 



Tide 



Cost Transactions 

Services for Individuals 

Services for Individuals 

Disability by County (Open/Closed) 

Source of Referral by. Disability 

Clients Rehabilitated by Disabil- 
ity, Occupation and Average Weekly 
Earnings (by County) 

Same as above (by District) 



VR4265 Counselor Master List Total 

VR4365 Supervised Cases Statistics 

VR613L0P Reason for Closure by Disability 
^VR61311P) Run first before Sort 

VR6565 Visual Impairments 

VR6575 Hearing Impairments 

VR6585 Orthopedic, Deformity 

VR6595 Amputation 

VR6605 Kental .and Personality Disorders 

VR6615 Keoplasms-, Allergic', Endocrine, 
Nervous Disorders, Heart & Cir- 
culatory 

■ VI 

v~R5625 Respiratory, £igcstivo, Conito* 
Urinary, Speech, Skin Conditions 



Run 

Frequency 



Annual 
Annual 
Annual 
Annual 
Bi-Annual 

Annual 

Annual 

Monthly 
Monthly 

Annual 

Annual , 

Annual 

Annual 

Annual 

Annual 

AnnuaV 
Annual 



Printout 
Distribution 



CO. ■ 

CO. , R.O., "D.C 
C.O.,R.O.,D.O. 
Evaluation 
Evaluat ion 

Evaluation 

Evaluation 

Evaluat ion 
Evaluation 

Evaluation 
Evaluation 
Evaluation 
Evaluat ion 
Evaluation 
Evaluation 

Evaluation 
Evaluation 
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■ ■/ ' 

Product Developed ' f 


Produc t 


Disi r lbuCion 


Dis^eininor ion 


Ft equency 


Cost by Report Transaction * 


CO. 


Annual 


Serviccc Provided to Individuals Report 


CO . , D .0 . 


j\ n n u a I 


Services Provided to Individuals Rcpor^ 


CO., D.O. 


Xnnua I 


Ad Hoc 


CO., D.O. 


M tfoc 


Referral Source by Disability / 


CO. 


Ad Hoc 


> * 

♦ 

Cnmnn t ( f ( vp /HnmoroaVp rs rehabilitations DY 
Disability by. County 


C.O.,R.O.,D.O. 


4 

Annual 


Clients Rehabilitated in Competitive Employ- 
ment by Disability ^ , 

District Productivity Report. 


CO. ,R„U. , U ,U. 


Annual »* * 


CO. 


Monthly , 


District Productivity Report 


CO. 


Monthly 


Reason" 1 for Closure by Disability Report • 


c n r n no 


V 

A nnu a I 


Ad Hoc ' * 


C.O, 


Ad Hoc 


Ad. Hoc * 


CO, 


Ad Hoc 


Ad Hoc 


CO. 


Ad Hoc 


Ad Hoc - 


C.O. 


Ad Hoc 


Ad Hoc 


C.O. 


Ad Hoc 


t M Hoc J 


C.O. 


Ad Hoc 


Ad Hpc , . 


CO. 


Ad Hoc 
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! Ru» No . 



1 



.iJMfehs 'in Statuses -by Kajor Uis- 
aMinfc„Condition w . 

Closed Cases by Fund Code and 
I/i* trice tiusbcr 
isontai Disabilities by Status 
(Opcri and Closed Cases) 
Services" Provided by Hrfjor Dis- ^ 
ablinfi Condition Code 
Total Cost for Ail Services .by 
•District (Rehab Facilities Cost, 
SSDI Coit, SSI Cost) 

Total 4 icnts * nd ToCal FisCal 
Year Ex^Kjdituras by Current 
-Status^witiiin Each Fund Coce by - 
County 

Total Clients and Total Fiscal 
Y^a'r Expenditures by Current 
^tatus within E3ch County 
Race within County*- Closed Cases* 

ssor - . ;; . • ;; „ 

• SSI 

Work Statuses at Time of Referral' 
for Closed Cases for Each District 
by Disability 

Closed Cases and Konths in Statue 
by Disability ' * t 

Cases Closed 28-30 with Closed 
frofa Status District • 



Uun 

Frequency 



Annual * 
Annual 

Annual * £ 

Annual > 

Annual 

Annual 

Annual 

Annual 

'Annual 
Annual 
Annual 



Yc incout 
Dis ir ibut ion 



Evaluation 
Evaluat ion 
Evaluation 
Evaluation 

Evaluation 



Evaluation 
Evaluation 
Evaluation 

s 

Evaluation < 

i 

Evaluation 
Evaluation 
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LA lfrtr 


Ad Hoc ' ^ 


:.o. 


Ad Hoc ^ 






Ad Hoc 


c.o. 


Ad Hoc 


Report on Client Served by County. CO. 


State Dept. 
f Comae rce 


Annual 

f 


Report on Client Served by County. C.O. 


State Dept. 
of Commerce 


Annual 


Ad Hoc 

Rc'port of Vages and Competitive Employment 


r ft-*' 
t 

State Dept. 
Off, of Budget 


'Ad' Hpc 
Annual 


Ad Hoc 


C.O. 


Ad Hoc 


Aa hoc 


C.O. 


Ad Hoc 
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0 





1 

Ti'» In 


Run 

Frequency , 


Printout 
Distribution 


Run No. 
VK6515 


Agc.'Nunbcr of Dependents, Karita) 
.?•••>..« Pri-ii«-v c ourcc of Support 
and Highest Grade of School Coin- 
cided on Cases Closed by 
District 


Annual 


Evaluation 


VR6525 


Sex, Previous Closure and Refer- 
ral Source Code by District 


Annual 


Evaluation 


VR654 5 


Public~Assis:ance Report at 
w*f>ml and Closure for Closed 
Cases 


Annual 


Evaluation 


VR6555 


Time on Public Assistance f Monthly 
Family Lncoaic, Spanish Surname 
and Trftal Nuc;b«r in Faaily by 
District for Closed Ca%es 


Annual 


Evaluat ion 


VR4555 


HchabiliTations anj! Cases Served 
of High Priority Target uroups 


Quarterly 


Evaluation ^ 

7r 

Fiscal and 
Evaluation 


VRCXE 


Nutrber-i** Cases Served in a Fis- 
cal Year where only a General 
i:edical has been Paid 


Annual 


VR6000^r 


Kando* Sample. Selection 


Ad HOc 


Evaluation 


VK60000I 


Randoa SatnpU Selection 


Ad Hoc 


Evaluation 
Evaluation 


VR61U0I 


Client Information Sheet 


Ad Hoc 


VR61235V 


^Status 26 Training Cases*' 
Vocational Coal Match 


Annual 


Evaluation * 


VH6000SI 


Randoa Sample Selection, 


Annual 


"Evaluation J 


vRsocoor 

- VR6 10271 


Randoa Sacple Selection 

Client Information Sheet Standard 


Annual 
Annual 


Evaluation r 
Evaluation^ 
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u i s l n ou i iu» 
Frequc ncv x 








<* 

Ad Hoc 


CO, 


Ad lt\c 


Ad Hoc 


CO. 
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. Ad Hoc 


CO. 


Ad Hoc 


Ad^Hoc , 
Ad Koc 


CO. 
CO. 


Ad Hoc 
• Adjloc 
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None 

Case Review Process^ 

Case Review Process ^ ^ 
J 

Case Review Process/ 
Federal Standard 2 


"None 

C.0. t R.0.,D.G 
Other State 

DVR Agencies, 
ftSA CO. 
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R.C, D.O. 

s 


Annual x 

1 . 

V Annual 
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Evaluation Follow-Up| 
Evaluation Follow-Up?* ? 
Evaluation Follow-UpJ 


RSA Central 0. 
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^Annual 
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l'r intone 

Di J U' Unit iou 


Product Developed 


Produc t 
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e 
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VH6535 


*1 

Work .Status - Referral! and 
Closure 

r ' * > 1 


Quarterly 


zy 1 

Evaluation 


District Office Profiles 
Annual* Report 


C.O.,D.O.,R.O 

c.o. t Dro.,R.o. 


Quarterly 
Annual 


VK68252* 


3 Work Staius, Source of Support, 
Earnings by Disability 


• Quarterly 


Evaluation 


District Office Profiles 
Annual Report C 


C.O.,C.O.,R,0. 

c.o. f o.o:,K.o. 


Quarterly 
Amui.il 


Vlt853l*2 


SSA-353 |<eport 

i i. i 


Monthly , 


Evaluation, 
District Office 






: . \ . • 




Pos* Employment * . 
: \ 


Monthly 


Evaluation 
District Office 


RSA-32 Report 


I'cdcral 
C.O.,D.O.,R.O. 


Annual 


VRSElttES 
> 


Similar Benefits 
" — : — — ■ j 


Monthly 


Evaluation 
District, Office 


Similar Benefits Reporc 


CO. 




VR6K80P 

■ * 


Delinquent Case Review 
1 — ^ — i — i _ 


Monthly 


Evaluation 
District Office 


Summary Report » ^ 


c.o,,a.o.,R.o. 


Monthly 
Quarterly 
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Management Information Review Procedure 



Obiective 



This procedure , is designed to guide in the analysis^and review of the 
current BVR Management Information System, 

f ' m 

Scope • 

The initial analysis undertaken will constitute the State reports manually 
prepared by the Evaluation Unit and disseminated throughout the Bureau, 
Procedure • 

I, ^J^ n °f Action * 

A, The Evaluation Staff will meet with each recipient and discuss 
each report and its potential utilization, Any changes or 
additions that might be tailored to the person 1 s p^ticular 
needs will also be discussed, J 

B, As a preliminary 'of visitations a meeting with two Regional < 
Administrators will be held, The purpose of the meeting is to 
obtain information to be used 1 for visits to the other Regional 

. * Offices, 

C, The same procedure will be used as a preliminary to visiting 
each District Office, Herei also, two District Offices will be 
visited, , , 

D, Upon completion of the preliminary visitations, meetings with 
the other Regions and .Districts will be arranged, Two District 
Offices ; will« be at each meeting, The meeting wi-li be attended 

* by the District Administrator and Assistant District Administrator, 

E, An agenda of the meetings will be supplied, either written or 
verbal, to each participant, A background of the Contract and MIS 
will be presented, - ^ ^ ' v - 
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F. 'When all District -and Regional Offices have 1 been contacted, 
information obtained will be""discussed with the Evaluation 
Section Staff, * " * 

G/ A plan for implementation of any changes, additions or deletions 
will be' developed", This plan will , include interaction with .the 
, user and ultimately will require, if needed, administrative 
approval, . • 

H. ( Implement t^e pla.n developed-, 

I. Conduct a follow-up survey of users. The survey will be used to 
assertain problem areas that may require further changes and 
impac-t of the plan, 

Information to be collected. 

A. How do they use the report? , -\ 

B. How often do they use it? 

C. Ideas on changes • , . 

D. Can we summarize data for them? 

E. What about Management by Exception? 

F. Do they disseminate' the report 7 

G. Do they want us to train them to read it? 

H. .Do they use- comparison of data from one year to the next?~ 

I. 'Po they use or cpuld they use one report in conjunction with another? 
Thfings to keep in mind > ' '• 

A. Pass cm information obtained from person to person. 

B. Might need another report 'to supplement currently received. " 
C jp nothing' else, this^ting will familiarize the person with 

the report and our interest. ' ' 

D, Don't discount any changes tha.t are not tailorable tb existing - * 
^ reports, Wo can add new reports, " 
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Procedure foft Request of' Information from Evaluation 
(Internal 4 and External) 



Steo 1 



Contact should be made Vith Evaluation and/or EDP for preliminary questions 
concerning a potential request for information. This informal session 
usually gives direction to the, person wakin^the request and familiarizes 
Evaluation and/or EDP with the potential reqpest. 



Stoo 2 



Written requests for information should be sent to the Administrator of ^ 
Administrative Services, with a copy to t|he Administrator of the Evaluation 
Section. The. nemo should contain a concise "description of the information 
needed!, the tine period covered, a time frame for completion of the report, 
and ^he reason for the request; ^ 



Stco 3 , 7* 



Based on tfcg reason given for the request, a 'decision will be made by 'the 
Administrator of Administrative Services "and Evaluation whether or not to 



\ 



.honor the request. 



Step 



The request will be analyzed by the Evaluation Section to determine if 
the information is currently available through existing reports. 



Step 5 



If the information is readily available and the time necessary to collect 
and report the data is cost-justified, - a* report will be generated- and 
forwarded to the requesting Section, The^pfocedure in this step could 
var*y depending on time available in the" Evaluation Section. Raw da„ta 
might be forwarded to the' requesting Unit for their own compilation. 

Step 6 

Ijc a computer run is necessary, a time and cost analysis will be conducted 
'by the Evaluation and/or EDP Uni^Cs) and*the results of this analysis will 
be reported to the Administrators of the Administrative Services and 
Evaluation Units' for their perusal. : . m 

Stcn 7 . * 

If approved^ in Step 5, a report, will be produced meeting the specifications 
outlined in S;ep 2, subject to any changes as a result of subsequent verbal 
or written communications. a 
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step r, 

In cither of the above situations, where the information is currently 
available or where 'a conouter run is needed to generate the data, the 
Evaluation Section is available to assist in the analysis of the data- 

Stcn 0 

All requests will be .entered in the Evaluation Request Log Book which 
will be used for reference and preparation of progress reports. 
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APPENDIX 44 

' PROCEDURE FOR REQUEST OF INFORMATION 
STEP 1 • , 

-Written requests for information should* be sent to the Administrator of Adminis- 
' trati-ve Services/ with a copy zo the' Administrator of the Evaluation Section. The 
memo should cont'ain a concise description of the information needed, the time 
period covered, a time frame for completion of the report, and the reason for the 
request. 

STEP 2 ' v ' * 

All written requests will be entered in the Evaluation Request Ix>g Book which will 

be used for .reference and preparation of progress reports.^ 

STEP 3 

Contact should be made with Evaluation" and/or >EDP for preliminary questions con^ 
cernir.g a potential -request f;r infor mat i on . — 

Guidelines' for Initi al Review of a Service Request 
n ? s ~~ ' : , 

" \ 'A. Are t the desired .resul-s.rcleariy specified by the requestor? 

'B. e A<ho - will use the res-its'? * * •* 

• ' ^ * * - ' *' * 

- ^ C- V?hat v wiljr the.resul% be used tfor? . ' . # \- 

- «» - ' ■ & " v 

' ' ' D. What',wd?ll be the fferuenc^ oVthis "request? How many copies are needed? ^ 

E. Are there alternative mestns to^gbtain^the data? . 

P. If, the data is not available,. sh6uld^it>^ c an EDP request or is it 

something we could do in the .Evaluation Section? - * ^ • 

G. What is the parity? - * n 

1. Director or Assistant Erector 

i , 

■* 2 Federal or State manda€es- or major agency objective . 

■» ■ • ■ ' ' 

3. Essential to u£=r 

4. Useful but not. essential to use*. (All out of agency requests for' - 
information will be -coded as priority 4) 

STE P 4- .- 

— : * *. » \ . 

Based on the- reason given for, the .request, a decision will be made by the Administrat 
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of Administrative Services and Evaluation whether pr not S to honor the request 



STEP 5 . " 

request will be analyzed by the Evaluation Section to determine if the infor- 
mation is currently available through existing reports. 

, STEP fc 6 > ' * 

jf the information is readily available and the time necessary to collect and seport 
the data is cost- justified, a report will be generated and forwarded, to the requesting 

Section. , , 

STEP,, 7 • ■ • . 

If a computer run is necessary," a time and cost analysis will be conducted by the . „ 
'Evaluation and/or EDP Unit(s) and the yefeults of this analysis will be reported to 
the Administrators of the Administrative Services and Evaluation Units for their 

perusal. * ' * . 

,The*<3uidelines to Review an EDP Request will bt applied at this £ime in the process. 

• 

STEP 8 ' , v 

If approved, an EDP report will be produced meeting the specifications 6utliried in ^ 
STFP 1, subject to any changes as a result "of subsequent verbal or written comuni- 
cations 

I 

• _ STEP 9 „ . - — .... 

in either of the above situations, where the information is currently available in 
the Evaluation 'unit or where a'computer run is needed to 'generate the data, the 
Evaluation Section is" available, to assist in the analysis of the data. 
S"TEP 10 - 

If the requestor wants the' Evaluation Unit to do an analysis of the data, and 

Evaluation Analysis Report -will be prepared. , ; , 

STEP ,11 , * ' 

l - • * 

All completed requests will be entered in the Evaluatipn Request Log Book which will- 
used for reference andpreparation of progress reportss. . . - 

i • , . c 
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FLOW DIAGRAM FOR; REQUEST OF INFORMATION FROM EVALUATION 
(latamal and External) ' , 
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GUIDELINES FOR INITIAL REVIEW OF A SERVICE REQUEST V 
<ll Are ttte desired results clearly specified by the requestor? 

If it is not clear what is wanted, ask questions to specifically define the type 
of information needed* Besure to address wkat information, what time period, in 
what fomat, and when is this information needed by- (date)? 

2. Who will use the results? * s t* * 

Is the person making the request doing so for someone else? If so, you may want 
^/ to suggest that the finished product of what will be obtained is reviewed by the 
intended user I before much effort is invested. Is the user of the requested 
information an employee of BVR? If not, will this outside source be willing to 
pay for this* informational request? 

3. • What will the results be used for? ^ 

The answer to^this question usually gives us a better feel" for priority a£ the 
request. # c 

4 • Vfoat will be the frequency of this request? How mapy copies are needed? 

To avoid unnecessary runs to produce extra copies and to justify the periadicai 
v cost of this request, ^it helps to get some indication of the number of cnpf«~s 

needed and the frequency needed. 
5 - Are there alternative means to obtain the data? * 

Consider the other data resources available. Would it be possible to answer the 

question from already existing evaluation reports or other special scequests*? 
6# - I£ the data is not available,' should it be an EDP request or is it some thing we 

' could do in the Evaluation Section? 

When making a- choice to do it in the Evaluation Section, consider the other work 
load we already have and the time it would t£»e to complete this request. Users 
should be encouraged to allow at least one week. ' if the request is .to be sub- 
mitted to EDP, assure that concerns from EDP guidelines area addressed.. 
"7- What i s the priority*? * v " 

— U ?A \. 

Refer the requestor to the priority scheme for a 1, 2, 3, or 4.. A "1" can be 

- - 530 • 
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-aNbssigned only by signature of Mr. Hagan, Mr- Kennedy, Mr. Latz or Dr. Bonner. 
A "2" is a due date not negotiable - it must be provided by the specified du^ 
date; example, a Federal report, or it is a, critical system correction or 
maintenance requirement, or it is essential for accomplishment ,of major agency 
objectives; example, case review process, similar benefits, independent lining, * 
deaf priority or agency goals in the number of rehabilitations. If the requestor** 
has the original document from which he is requesting the change - i.e. RSA- 
PI changing SGA to $300 per month, ask .for a copy of the document to verify the 
due dace. A "3" is essential to the user as it results in loss of effectiveness 
or efficiency, and a ,! 4 M .is useful but not essential to the user. All out of 
agency requests fo£ information will "be coded as a "4". 
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' GU-jpSLINES TO REVIEW AN EDP REQUEST 

\y'lt the-recuest does Vequire EDP services, consider the type of EDP request. 

/-Each of the following types Raises special questions • 
/ ' % " 

' • lm . g ha ^e -conouter ,file constants .such as counselor numbers, "fiscal budgets, 

' rer.ab goals. • » 

a. , Has th^chan'ge(s) authorized by the appropriate person? ' 

b. Should anyone else be notified?'' . 
. ^ c. .what turn around evidence of change is* needed? ' \ 

2- Run an existing proara, with a modification such as change in date period or 
population. ^ 

4 a. Is the change clearly identified? , 

• ' b. Is the data to be comparable to any other data? 
" : 3 * - ^ an ^£in2, °ro«r am by changing something not anticipated in the original 
request, such as a change .in sort order or added elements, 
a. 1\ the change clearly identified? ' 

b- is the data to be, comparable to any other data? ' ' 

,4- ^eFare a n^nrog^am to.retrieve 'specified elements in a requested format ' 
. . < (not anticipated to become a' routine 'report)'. " ' 

a. Which elements are wanted? 'dearly defined? ' 

b. Is format specified? 

-c. could an existing report'with modifications, providers data? 
t d. Do .we want flexibility to rerun with specific changes? , 
5. Program a new rer^ort: to become routine . 

• a. Which data 'elements are needed? ' \ 4 

b - Is format specified? - 1 

c. what will distribution schedule >e? 

d. Is it to be comparable to other data? 

e. Will user training be required? ' who? # When? . ' 
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Add new file element(s) to the computer files* 

a. .What input process has been established? 

b! Have all .parties involved in input been consulted? 

c. Are any 'forms or instructions changeT^n^ree^sary? Training? 

d. Is there a clear documented definition of the element? 

e. Does this element make some other element unnecessary? 

f . Will we try to retroactively collect the new file element? 
Delete an existing file element frop the computer files. 

a. What is it being used for? 

b. Have all users been consulted? 

c. ^ What other reports/systems may be affected? 

d. Has the input vehicle been eliminated? 

e. Should the £Lle retain what it has? 

Change an existing internal system pi processing data. 

a. What are the current uses?* 

b. Have all users been consulted? 

c. Is the change clearly defined? 

d. is the justification documented? 

e. What problems could ^Lt create? 
Establish a- new internal 'system of processing data. 
3.>*Could a -modification to existing" systems suffice? 
b. is "the process, clearly defined? # h V 
c- Is the justification documented? 

d. Have potential users all been consulted? 

e. Will the results overlap with other data-' collection systems 

Combination of any of. the above 1 through 9. 

% 

* * • 
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REVISED 

PENNSYLVANIA BUREAU OF VOCATIONAL REHABILITATION 
EVALUATION REQUEST LOG 
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•4 V 
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' COMMONWEALTH- OF PENNSYLVANIA ' 
DEPARTMENT OF LABOR AND INDUSTRY 
BUREAU OF VOCATIONAL REHABILITATION 



Memo No.* 81-450. IS 



March 26, 1981 



Planning Branch 
Evaluation Memorahdum 



Subject: Informational Request Procedure 



From: . Harry W. Guise 
„ - Administrator 



/ 



Central Office and district office staffs' require a variety of types of informa- / 
tion for their own needs and for requests of many sources outside *6ur Agency. 

In an attempt to respond to these requests, several factors must be considered; 
namely, time, cost analysis/ and availability of information. These considera- 
tions are especially important fox sources outside our Agency since a fee /Will 
be charged for .any information. - , / 

Jo consolidate this responsibility for informational requests and to^ssist in 
obtaining the necessary information , in. a timely manner, the following prpcediires 
are being implemented; V / * ' 



/ 



Step 1 - Verbal contact should be made wifrl^the Program Evaluation Section 
, concerning a potential request" of information. This contact can- 
give direction. to the jperson making the request f and familiarize 
the* Program Evali^ti§p Section with the potential request. 

Step 2 - Following the verbal' contact, a request will be made, in writing 
arid" sent in the followirtg manner: 

- - -X 

A. Central Office staff - All requests will be sent to x the 
Administrator of "Program Evaluation. 

* » • 

B. District and regional staff .- The respective administrator , 
will' send request to the Assistant Director of Field / 
Operations with a copy to the Administrator of Program 
Evaluation. 



NOTE: 



C. Information requests from sources outside our Agency via the 
district &r regional office - The district or regional' 
administrator will notify the individual requesting the 
information to" write a letter to the Administrator of Program 
Evaluation. • 

THE ABOVE PROCEDURES DO NOT PERTAIN TO THE PROGRAM EVALUATION SECTION 
LIBRARY REQUESTS^ 



V 



Resulting Action: 
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Effective Date Immediately 
Distribution A 5 Supervisors 

^ 

P § P Approval _JMM 



Evaluation Memorandum 



Memo No, 



81-4*50.15 



Each request will__contain the following: «£- . / 

' ' • c " . . ;' •• 

x JL Concise description of information needed* • , 

* " * 2, Time period covered. ± « v 

3. Date information needed, t / 

4, Reason for request: 

a. Purpose of dnformation^. 
<, b. Who will use information. 

When the request is received by the Program Evaluation Section, it 
will be recorded in a logbook. This log will be used as a record 
of lOte requests «and also for final disposition of the requests- 

a »" » * 

Step 3 : - Based on the information provided in the request, a decision 

whether or not to proceed will be made by the appropriate Central 
Office staff. ; * . ' 



Step 4 



If it is decided to deny the request, the person making the 
request will be informed;* For, sources outsidg our Agency, the 
district and the requester will be notified d¥ the decision by 
the Program Evaluation Section," 



Step 5 - If the request is approved, the Program Evaluation Section will 
i- determine the availability ancf the cost of Collecting the 

• , information. For ^sources ^outside our Agency, the district 

and requester will be notified of the decision by the Program . 
Evaluation Section. 

Step 6 - If the information requested is- already available,, a report 
will be prepared and Sent to the* person faking the request. 
, For sources outside our Agency, a fee for staff time will be 
charged.- v • 

Step! -If the information requested is not, readily available, a computer 
program will be written. After the information is received, a 
^ report will be prepared and sent tp the person making the request. 

For sources -outside -our Agency, a feet for staff time and computer 
costs will be*. charged; 

This information should be relayed at least verbally to all your professional 
staff. Duplication on your own equipment and further distribution will be t 
according to your own discretion. *' ' 
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APPENDIX 49 

Pennsylvania 1 3 Perspective on its Involvement in the^ Federal Department 1 
of Education's Model Evaluation Contract, 1978 to 1981 

— ; • 

Presented to April 1981 Conference of, the American Personnel & Guidance 
Association, St. Louis, MO by Donald E. Hossler, Program Analyst. 

Pennsylvania has, is, and will continue to benefit from its participation * 
in the National Model Evaluation Unit Federal Contract with the U. S. 
Department of Educatfibn. ' 

The Program Evaluation Section of tne PA Bureau'of Vocational Rehabilitation r 
begancontract Work activity 1q October 1978', contract fitading concludes in 
September 1981.. We hav£ developed several products from a variety of tasks 
that have helped us and should prove helpful <to other social service agencies \ 
and personnel in tfie field of program evaluation. This handout describes v 
briefly our contract work.. We encourage you to contact us for detailed infor- 
mation. Telephone or write to: Harry W. Guise, Administrator of Progr&h ^ * 
Evaluation, PA Bureau of Vocational Rehabilitation, Labor & Industry Building, *| 
Seventh and Forster Streets, Room 1317, Harrisburg, PA 17120 - (717)787-5123. ./ ^ 

Contract work activity has required technical assistance from^a variety of « i 
public and "private sources. Tracking and monitoring systems have* been ref in$d x ' 
and seyeral new dissemination and utilization' strategies. were utilized. Th^/^ 
field testing of the program evaluation standards and a Facilities I^iformatioi^ 5 ' 
System will be completed when the contract terminates. ' ' X,' . * 

Management Information System - A review and assessment of Bureau utilization 

of materials disseminated by the Program Evaluation Sepfcion is being completed * . 

Case Review Process - Continued development and implementation of reviews, that 
arte concerned with case service patterns within the, rehabilitation process and 
provide information for use in program monitoring. ~ 

Client' Follow-up Studies - Utilized to determine^how well the rehabilitation ■ . 
process is meeting client nee'ds and Federal Program EValtf&£ ion Standards. 

♦ *** 

Revision of Profile Analysis Technique (PAT) - This,is # a method of displaying 

aggregate data for analysis and evaluation purposes. * . 

Pilot Study for Collection, of Similar Benefits - This is a two-phased system 
to facilitate cpllection & documentation from counselor to management via > 
data processing. ^ ( • 

Technical Assistance Center (TAC) .- This information system responds to a 
variety of r.equest^ from management and staff, in'the field- and our Central 
Office. The information available ranges from medical aspects of disability 
to principles of management,. ' "The Rehab Brief", published by the University 
of Florida RRI, is a valuable vehicle utilized in assessing and meeting Bureau 
needs. f * 

"Although funding for the contract. will end Saon, PA will continue to refine 
and develop many of the contract era work activities in program evaluation. 
If you desire more information or have questions, telephone or write us. 
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